Compilation from macOS(Intel) to linux x86_64 finaly works. Cross compilation of the rust code with bazel. Part 2.

This is the continuation of cross-compilation  topic that I've started in previous part

What? Do I really need to configure C++ compiler to compile rust? Cross compilation of the rust code with bazel. Part 1.

Since we need to add C++ cross-compilation toolchain, let's add the pure C++ project.
This will help us to test the C++ toolchain separately

I've added a typical C++ hello world impliemntaion and a BUILD file for it:

cc_binary(
name = "hello-world",
srcs = ["main.cpp"],
)

And our goal is to make it compilable for linux by command:

bazel build //cpp/hello_world:hello-world --platforms=//:x86_64-linux --incompatible_enable_cc_toolchain_resolution

If we just try it, we get and error, 

(23:01:17) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @baze
l_tools//tools/cpp:current_cc_toolchain: Unable to find a CC toolchain using toolchain resolution. Did you properly set --platforms?                                
(23:01:17) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@bazel_tools
//tools/cpp:current_cc_toolchain' failed                                                                                                                            
(23:01:17) ERROR: Analysis of target '//cpp/hello_world:hello-world' failed; build aborted:

There are a couple of articles about setting up the C++ toolchain. 

At the moment some of the articles refer to the old way of selecting the c++ compilation toolchain, throught the "--crosstool_top" option. As far as I've understood, it is obsolete now. In the right way the selection of toolchain should be based on platforms. So some parts of the articles could be redundant.

Interesting part, that is not described in  articles is where to get the toolchains themself. For the quick start I'll try to use the precompiled mac toolchains from the project macos-cross-toolchains.

I'll skip the logic that would select the file for dowloading and  download the toolchain, for the time being I'll just vendor the toolchain into my repository.

Short video with explanation of basic terms of the toolchain:
https://www.youtube.com/watch?v=5ugdNGVjSRQ

So we need to create a "c++ toolchain", for our downloaded repostiory.
Let's start from

cc_toolchain(
name = "x86_64-linux-musl_toolchain",
toolchain_config = ":x86_64-linux-musl_toolchain_config",
# We tell the toolchain, that for every command, all the files are needed
# This is not ideal, but simple.
all_files = ":all",
compiler_files = ":all",
dwp_files = ":all",
linker_files = ":all",
objcopy_files = ":all",
strip_files = ":all",
)

The ":all" is a file group that includes all the files, that are downloaded.

And the the ":x86_64-linux-musl_toolchain_config" is a stub for now:

filegroup(
name = "x86_64-linux-musl_toolchain_config",
)

filegroup(
name = "all",
srcs = glob(["**/**"]),
)

We didn't specify anywhere, that the toolchain is for the specific target CPU and os. To do so wee need another toolchain layer, that is called "toolchain".

toolchain(
name = "x86_64-linux-musl",
toolchain = ":x86_64-linux-musl_toolchain",
target_compatible_with = [
"@platforms//os:linux",
"@platforms//cpu:x86_64"
],
toolchain_type = "@rules_cc//cc:toolchain_type",
)

If we try to compile now, it still fails to find the cc toolchain:

(09:38:07) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @bazel_tool
s//tools/cpp:current_cc_toolchain: Unable to find a CC toolchain using toolchain resolution. Did you properly set --platforms?                                            
(09:38:07) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@bazel_tools//tool
s/cpp:current_cc_toolchain' failed

It is because we didn't register it in the bazel workspace. We do it in the WORKSPACE file:

register_toolchains(
"//toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl"
)

Now the toolchain is found, and the problem is that the toolchain configuration, that we provided has  a wrong type.

(09:40:11) INFO: Current date is 2023-09-03                                                                                                                               
(09:40:15) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/BUILD.bazel:10:13: in toolchain_config attribute of cc_
toolchain rule //toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolchain: '//toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolcha
in_config' does not have mandatory providers: 'CcToolchainConfigInfo'                                                                                                     
(09:40:15) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/BUILD.bazel:10:13: Analysis of target '//toolchains/thi
rdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolchain' failed

We need to create the toolchain configuration. The documentation of how to do it is quite fragmental, and often obsolete. In the video above a good example is provided. Let's try to use it.

There are shortcuts for different host platforms in the "bazel_tools" repository. On mac and linux we can use "unix_cc_toolchain_config".

load("@bazel_tools//tools/cpp:unix_cc_toolchain_config.bzl", "cc_toolchain_config") 

cc_toolchain_config(
name = "x86_64-linux-musl_toolchain_config",
abi_libc_version = "unknown",
abi_version = "unknown",
compiler = "gcc",
cpu = "x86_64",
host_system_name = "local",
target_libc = "unknown",
target_system_name = "x86_64-linux-musl",
toolchain_identifier = "x86_64-linux-musl",
tool_paths = {
"ar": "ar",
"cpp": BIN_PREFIX + "g++",
"gcc": BIN_PREFIX + "gcc",
"dwp": "dwp",
"gcov": "gcov",
"ld": "ld",
"nm": "nm",
"objcopy": "objcopy",
"objdump": "objdump",
"strip": "strip",
"llvm-cov": "llvm-cov",
}, 
)

The bunch of parameters, that are required in the call above, seems to be obsolete at the moment, so just providing "unknwon" there.

If we try to build now, we are getting the new error:

(10:19:36) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/cpp/hello_world/BUILD.bazel:1:10: Compiling cpp/hello_world/main.cpp failed: undeclared inclusion(s) in
 rule '//cpp/hello_world:hello-world':                                                                                                                                    
this rule is missing dependency declarations for the following files included by 'cpp/hello_world/main.cpp':                                                              
  '/Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/stdc-predef.h'          
  '/Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/iostream'  

To fix it we need to tell compiler, that these directories are "system includes".

compile_flags = [
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/x86_64-unknown-linux-musl",
]

Compilation now fails with other problem:

main.cpp:(.text+0x11): undefined reference to `std::cout'

We need to link our binary with the standard library, this we do by specifying '-lstdc++' for linker arguments. By the way, last time we just defined the right paths only for compilers, let's fix it for all the tools

BIN_PREFIX = "bin/x86_64-unknown-linux-musl-"

cc_toolchain_config(
name = "x86_64-linux-musl_toolchain_config",
abi_libc_version = "unknown",
abi_version = "unknown",
compiler = "gcc",
cpu = "x86_64",
host_system_name = "local",
target_libc = "unknown",
target_system_name = "x86_64-linux-musl",
toolchain_identifier = "x86_64-linux-musl",
tool_paths = {
"ar": BIN_PREFIX + "ar",
"cpp": BIN_PREFIX + "g++",
"gcc": BIN_PREFIX + "gcc",
"dwp": BIN_PREFIX + "dwp",
"gcov": BIN_PREFIX + "gcov",
"ld": BIN_PREFIX + "ld",
"nm": BIN_PREFIX + "nm",
"objcopy": BIN_PREFIX + "objcopy",
"objdump": BIN_PREFIX + "objdump",
"strip": BIN_PREFIX + "strip",
"llvm-cov": BIN_PREFIX + "llvm-cov",
},
compile_flags = [
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/x86_64-unknown-linux-musl",
],
link_libs = [
"-lstdc++",
],
)

Now we can build our C++ application:

 % bazel build //cpp/hello_world:hello-world --platforms=//:x86_64-linux --incomp
atible_enable_cc_toolchain_resolution -s --sandbox_debug                                                                                 
(14:54:12) INFO: Invocation ID: c0a5ebf7-eeae-4564-b927-a909d31df854                                                                     
(14:54:12) INFO: Current date is 2023-09-03                                                                                              
(14:54:12) INFO: Analyzed target //cpp/hello_world:hello-world (0 packages loaded, 0 targets configured).                                
(14:54:12) INFO: Found 1 target...                                                                                                       
Target //cpp/hello_world:hello-world up-to-date:                                                                                         
  bazel-bin/cpp/hello_world/hello-world

And it is a Linux binary:

 % file bazel-bin/cpp/hello_world/hello-world                                    
bazel-bin/cpp/hello_world/hello-world: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-
x86_64.so.1, not stripped

To test it I used docker:

 % docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu   
root@88f89297cb6c:/# cd /ext                                                                                                             
root@88f89297cb6c:/ext# ls -l bazel-bin/cpp/hello_world/hello-world                                                                      
-r-xr-xr-x 1 root root 16448 Sep  3 12:50 bazel-bin/cpp/hello_world/hello-world                                                          
root@88f89297cb6c:/ext# ./bazel-bin/cpp/hello_world/hello-world                                                                          
bash: ./bazel-bin/cpp/hello_world/hello-world: No such file or directory

So my binary exists, but fails to run. It looks like it could be symlink that can not be resolved in docker, but after closer look, it turns out, that the libc.so can not be found by the linker:

root@88f89297cb6c:/ext# ldd bazel-bin/cpp/hello_world/hello-world                                                                        
        linux-vdso.so.1 (0x00007ffdefd66000)                                                                                             
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f48484ae000)                                                      
        libc.so => not found                                                                                                             
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f48483c7000)                                                                
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f484819f000)                                                                
        /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f48486de000)                                                     
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f484817f000)

We use the musl toolchain, but it is not available in the container as a shared library.

MUSL gcc implementation allows static linking of executable. So this can be acheived by setting

features = [
"fully_static_link",
]

attribute to the binary.

After adding, it the binary is not dynamically linked any more and ldd, fails with an error:

% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu ldd /ext/bazel-bin/cpp/hello_world/
hello-world                                                                                                                                                               
        not a dynamic executable  

But we can run it under linux now:

 % docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu /ext/bazel-bin/cpp/hello_world/hell
o-world                                                                                                                                                                   
Hello, World!

Does rust works now?

Let's test what is the status of our rust example.
% bazel build //rust/pure_rust_example:hello_world --platforms=//:x86_64-linux  --incompatible_enable_cc_toolchai
n_resolution                                                                                                                                                              
(15:45:31) INFO: Invocation ID: d9f2d179-8cba-4136-855c-0b9237bee43c                                                                                                      
(15:45:31) INFO: Current date is 2023-09-03                                                                                                                               
(15:45:31) INFO: Analyzed target //rust/pure_rust_example:hello_world (0 packages loaded, 0 targets configured).                                                          
(15:45:31) INFO: Found 1 target...                                                                                                                                        
Target //rust/pure_rust_example:hello_world up-to-date:                                                                                                                   
  bazel-bin/rust/pure_rust_example/hello_world  
 WOW, it compiles now!!!
Let's test if it works:
% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu /ext/bazel-bin/rust/pure_rust_example/hello_world                                                                                                                                                            
Hello, world!
It does, and how does it link?
% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu ldd /ext/bazel-bin/rust/pure_rust_example/hello_world                                                                                                                                                        
        statically linked
It is a statitc binary. 

Conclusion

 We managed to cross-compile a static binary for c++ and rust. For now we only support only one target and one host system. Another problem is that the whole toolchain is added to the git-repository.
Next steps would be:
  • download the toolchains dynamically
  • support more target platforms
  • make our example project more complex, to ensure it works in other cases too
The state of the project at the end of this chapter is in github.
 



Comments

Post a Comment

Popular posts from this blog

Cross compilation for ARM v7. Cross compilation of the rust code with bazel. Part 3.

This is the third part of the series "Cross compilation of rust with bazel". Previous topics: What? Do I really need to configure C++ compiler to compile rust? Cross compilation of the rust code with bazel. Part 1. Compilation from macOS(Intel) to linux x86_64 finaly works. Cross compilation of the rust code with bazel. Part 2.   Next challenge is to cross-compile for armv7. This part we did together with my colleague Dima and we did it on linux. First thing, that we need is to add the target platform into our root BUILD.bazel platform ( name = "armv7-linux-musleabihf" , constraint_values = [ "@platforms//cpu:armv7" , "@platforms//os:linux" , ], ) And rust toolchain for the new target in the WORKSPACE file: rust_register_toolchains ( edition = "2021" , extra_target_triples = [ "x86_64-unknown-linux-musl" , "armv7-unknown-linux-musleabihf" , # NEW ], ) And as we remember from the part 1 ,...
Read more

Scaling number of supported platforms. Cross compilation of the rust code with bazel. Part 5

Image
  This is the 5th part of Basel cross-compilation series. Previous posts: What? Do I really need to configure C++ compiler to compile rust? Compilation from macOS(Intel) to linux x86_64 finaly works.  Cross compilation for ARM v7.  Fetching toolchains at build time.   All the build files in the previous steps looked pretty much the same, the only differences were: URLs exec architecture target architecture some compiler flags (like paths) Of cause we don't want to support a separate build file for every host and target combination. The way around it maybe to generate it from the template. The easiest way would be to use string formatting int the *.bzl file. So we create `toolchains/cpp_toolchains.bzl`, and store the BUILD file content in the variable there: BUILD_FILE_TEMPLATE = """ load("@bazel_tools//tools/cpp:unix_cc_toolchain_config.bzl", "cc_toolchain_config") load...
Read more