Compilation from macOS(Intel) to linux x86_64 finaly works. Cross compilation of the rust code with bazel. Part 2.

This is the continuation of cross-compilation  topic that I've started in previous part

What? Do I really need to configure C++ compiler to compile rust? Cross compilation of the rust code with bazel. Part 1.

Since we need to add C++ cross-compilation toolchain, let's add the pure C++ project.
This will help us to test the C++ toolchain separately

I've added a typical C++ hello world impliemntaion and a BUILD file for it:

cc_binary(
name = "hello-world",
srcs = ["main.cpp"],
)

And our goal is to make it compilable for linux by command:

bazel build //cpp/hello_world:hello-world --platforms=//:x86_64-linux --incompatible_enable_cc_toolchain_resolution

If we just try it, we get and error, 

(23:01:17) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @baze
l_tools//tools/cpp:current_cc_toolchain: Unable to find a CC toolchain using toolchain resolution. Did you properly set --platforms?                                
(23:01:17) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@bazel_tools
//tools/cpp:current_cc_toolchain' failed                                                                                                                            
(23:01:17) ERROR: Analysis of target '//cpp/hello_world:hello-world' failed; build aborted:

There are a couple of articles about setting up the C++ toolchain. 

At the moment some of the articles refer to the old way of selecting the c++ compilation toolchain, throught the "--crosstool_top" option. As far as I've understood, it is obsolete now. In the right way the selection of toolchain should be based on platforms. So some parts of the articles could be redundant.

Interesting part, that is not described in  articles is where to get the toolchains themself. For the quick start I'll try to use the precompiled mac toolchains from the project macos-cross-toolchains.

I'll skip the logic that would select the file for dowloading and  download the toolchain, for the time being I'll just vendor the toolchain into my repository.

Short video with explanation of basic terms of the toolchain:
https://www.youtube.com/watch?v=5ugdNGVjSRQ

So we need to create a "c++ toolchain", for our downloaded repostiory.
Let's start from

cc_toolchain(
name = "x86_64-linux-musl_toolchain",
toolchain_config = ":x86_64-linux-musl_toolchain_config",
# We tell the toolchain, that for every command, all the files are needed
# This is not ideal, but simple.
all_files = ":all",
compiler_files = ":all",
dwp_files = ":all",
linker_files = ":all",
objcopy_files = ":all",
strip_files = ":all",
)

The ":all" is a file group that includes all the files, that are downloaded.

And the the ":x86_64-linux-musl_toolchain_config" is a stub for now:

filegroup(
name = "x86_64-linux-musl_toolchain_config",
)

filegroup(
name = "all",
srcs = glob(["**/**"]),
)

We didn't specify anywhere, that the toolchain is for the specific target CPU and os. To do so wee need another toolchain layer, that is called "toolchain".

toolchain(
name = "x86_64-linux-musl",
toolchain = ":x86_64-linux-musl_toolchain",
target_compatible_with = [
"@platforms//os:linux",
"@platforms//cpu:x86_64"
],
toolchain_type = "@rules_cc//cc:toolchain_type",
)

If we try to compile now, it still fails to find the cc toolchain:

(09:38:07) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @bazel_tool
s//tools/cpp:current_cc_toolchain: Unable to find a CC toolchain using toolchain resolution. Did you properly set --platforms?                                            
(09:38:07) ERROR: /private/var/tmp/_bazel_evgenypetrov/e83d964deb47c21842622d1b7ffa55fc/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@bazel_tools//tool
s/cpp:current_cc_toolchain' failed

It is because we didn't register it in the bazel workspace. We do it in the WORKSPACE file:

register_toolchains(
"//toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl"
)

Now the toolchain is found, and the problem is that the toolchain configuration, that we provided has  a wrong type.

(09:40:11) INFO: Current date is 2023-09-03                                                                                                                               
(09:40:15) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/BUILD.bazel:10:13: in toolchain_config attribute of cc_
toolchain rule //toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolchain: '//toolchains/thirdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolcha
in_config' does not have mandatory providers: 'CcToolchainConfigInfo'                                                                                                     
(09:40:15) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/BUILD.bazel:10:13: Analysis of target '//toolchains/thi
rdparty/x86_64-unknown-linux-musl:x86_64-linux-musl_toolchain' failed

We need to create the toolchain configuration. The documentation of how to do it is quite fragmental, and often obsolete. In the video above a good example is provided. Let's try to use it.

There are shortcuts for different host platforms in the "bazel_tools" repository. On mac and linux we can use "unix_cc_toolchain_config".

load("@bazel_tools//tools/cpp:unix_cc_toolchain_config.bzl", "cc_toolchain_config") 

cc_toolchain_config(
name = "x86_64-linux-musl_toolchain_config",
abi_libc_version = "unknown",
abi_version = "unknown",
compiler = "gcc",
cpu = "x86_64",
host_system_name = "local",
target_libc = "unknown",
target_system_name = "x86_64-linux-musl",
toolchain_identifier = "x86_64-linux-musl",
tool_paths = {
"ar": "ar",
"cpp": BIN_PREFIX + "g++",
"gcc": BIN_PREFIX + "gcc",
"dwp": "dwp",
"gcov": "gcov",
"ld": "ld",
"nm": "nm",
"objcopy": "objcopy",
"objdump": "objdump",
"strip": "strip",
"llvm-cov": "llvm-cov",
}, 
)

The bunch of parameters, that are required in the call above, seems to be obsolete at the moment, so just providing "unknwon" there.

If we try to build now, we are getting the new error:

(10:19:36) ERROR: /Users/evgenypetrov/work/bazel_cross_compilation/cpp/hello_world/BUILD.bazel:1:10: Compiling cpp/hello_world/main.cpp failed: undeclared inclusion(s) in
 rule '//cpp/hello_world:hello-world':                                                                                                                                    
this rule is missing dependency declarations for the following files included by 'cpp/hello_world/main.cpp':                                                              
  '/Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/stdc-predef.h'          
  '/Users/evgenypetrov/work/bazel_cross_compilation/toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/iostream'  

To fix it we need to tell compiler, that these directories are "system includes".

compile_flags = [
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/x86_64-unknown-linux-musl",
]

Compilation now fails with other problem:

main.cpp:(.text+0x11): undefined reference to `std::cout'

We need to link our binary with the standard library, this we do by specifying '-lstdc++' for linker arguments. By the way, last time we just defined the right paths only for compilers, let's fix it for all the tools

BIN_PREFIX = "bin/x86_64-unknown-linux-musl-"

cc_toolchain_config(
name = "x86_64-linux-musl_toolchain_config",
abi_libc_version = "unknown",
abi_version = "unknown",
compiler = "gcc",
cpu = "x86_64",
host_system_name = "local",
target_libc = "unknown",
target_system_name = "x86_64-linux-musl",
toolchain_identifier = "x86_64-linux-musl",
tool_paths = {
"ar": BIN_PREFIX + "ar",
"cpp": BIN_PREFIX + "g++",
"gcc": BIN_PREFIX + "gcc",
"dwp": BIN_PREFIX + "dwp",
"gcov": BIN_PREFIX + "gcov",
"ld": BIN_PREFIX + "ld",
"nm": BIN_PREFIX + "nm",
"objcopy": BIN_PREFIX + "objcopy",
"objdump": BIN_PREFIX + "objdump",
"strip": BIN_PREFIX + "strip",
"llvm-cov": BIN_PREFIX + "llvm-cov",
},
compile_flags = [
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/",
"-isystem", "toolchains/thirdparty/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/include/c++/11.2.0/x86_64-unknown-linux-musl",
],
link_libs = [
"-lstdc++",
],
)

Now we can build our C++ application:

 % bazel build //cpp/hello_world:hello-world --platforms=//:x86_64-linux --incomp
atible_enable_cc_toolchain_resolution -s --sandbox_debug                                                                                 
(14:54:12) INFO: Invocation ID: c0a5ebf7-eeae-4564-b927-a909d31df854                                                                     
(14:54:12) INFO: Current date is 2023-09-03                                                                                              
(14:54:12) INFO: Analyzed target //cpp/hello_world:hello-world (0 packages loaded, 0 targets configured).                                
(14:54:12) INFO: Found 1 target...                                                                                                       
Target //cpp/hello_world:hello-world up-to-date:                                                                                         
  bazel-bin/cpp/hello_world/hello-world

And it is a Linux binary:

 % file bazel-bin/cpp/hello_world/hello-world                                    
bazel-bin/cpp/hello_world/hello-world: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-
x86_64.so.1, not stripped

To test it I used docker:

 % docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu   
root@88f89297cb6c:/# cd /ext                                                                                                             
root@88f89297cb6c:/ext# ls -l bazel-bin/cpp/hello_world/hello-world                                                                      
-r-xr-xr-x 1 root root 16448 Sep  3 12:50 bazel-bin/cpp/hello_world/hello-world                                                          
root@88f89297cb6c:/ext# ./bazel-bin/cpp/hello_world/hello-world                                                                          
bash: ./bazel-bin/cpp/hello_world/hello-world: No such file or directory

So my binary exists, but fails to run. It looks like it could be symlink that can not be resolved in docker, but after closer look, it turns out, that the libc.so can not be found by the linker:

root@88f89297cb6c:/ext# ldd bazel-bin/cpp/hello_world/hello-world                                                                        
        linux-vdso.so.1 (0x00007ffdefd66000)                                                                                             
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f48484ae000)                                                      
        libc.so => not found                                                                                                             
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f48483c7000)                                                                
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f484819f000)                                                                
        /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007f48486de000)                                                     
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f484817f000)

We use the musl toolchain, but it is not available in the container as a shared library.

MUSL gcc implementation allows static linking of executable. So this can be acheived by setting

features = [
"fully_static_link",
]

attribute to the binary.

After adding, it the binary is not dynamically linked any more and ldd, fails with an error:

% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu ldd /ext/bazel-bin/cpp/hello_world/
hello-world                                                                                                                                                               
        not a dynamic executable  

But we can run it under linux now:

 % docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu /ext/bazel-bin/cpp/hello_world/hell
o-world                                                                                                                                                                   
Hello, World!

Does rust works now?

Let's test what is the status of our rust example.
% bazel build //rust/pure_rust_example:hello_world --platforms=//:x86_64-linux  --incompatible_enable_cc_toolchai
n_resolution                                                                                                                                                              
(15:45:31) INFO: Invocation ID: d9f2d179-8cba-4136-855c-0b9237bee43c                                                                                                      
(15:45:31) INFO: Current date is 2023-09-03                                                                                                                               
(15:45:31) INFO: Analyzed target //rust/pure_rust_example:hello_world (0 packages loaded, 0 targets configured).                                                          
(15:45:31) INFO: Found 1 target...                                                                                                                                        
Target //rust/pure_rust_example:hello_world up-to-date:                                                                                                                   
  bazel-bin/rust/pure_rust_example/hello_world  
 WOW, it compiles now!!!
Let's test if it works:
% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu /ext/bazel-bin/rust/pure_rust_example/hello_world                                                                                                                                                            
Hello, world!
It does, and how does it link?
% docker run -v `pwd`:/ext  -v /private/var/tmp/:/private/var/tmp/ -it ubuntu ldd /ext/bazel-bin/rust/pure_rust_example/hello_world                                                                                                                                                        
        statically linked
It is a statitc binary. 

Conclusion

 We managed to cross-compile a static binary for c++ and rust. For now we only support only one target and one host system. Another problem is that the whole toolchain is added to the git-repository.
Next steps would be:
  • download the toolchains dynamically
  • support more target platforms
  • make our example project more complex, to ensure it works in other cases too
The state of the project at the end of this chapter is in github.
 



Comments

Post a Comment

Popular posts from this blog

Левиафан

Image
Photo from Wikipedia article Пару месяцев назад у меня с коллегами была дискуссия: допустимо ли качать видео из интернета. Я выступал против скачивания (с заведомо нелегальных ресурсов). Одним из ключевых аргументов моих оппонентов было: "Вот "Левиафан" запретят сейчас в России и вообще его никак не посмотреть в оригинале будет". Сегодня я посмотрел "Левиафана" в кино в Берлине русском языке. И похоже без купюр: в паре сцен мата было столько, что неуютно было. Начитавшись отзывов в интернете, я ожидал от фильма истории про этакого Бэтмена из российской глубинки. Который в одиночку прёт против системы, вскрывая все её темные стороны. Ну, а за что еще запрещать фильм? Что же увидел я? В течение всего фильма происходила тяжба двух "хозяйствующих субъектов". Один хотел землю выкупить под "государственные нужды". Другой, насколько я понимаю, был в общем то не против, весь вопрос в цене был. Лично мне трудно понять кто из них прав был
Read more

Key West, FL

Image
Продлив свой день на 5 часов (именно такая разница во времени в Берлине и Майами), мы вышли из самолета в аэропорту Майами. Потратили около трех часов в очередях: на паспортный контроль, в таможню и, внезапно, в офисе проката автомобилей. Взяли свой красный форд мустанг за $40 в день и двинулись на юг по дороге US1 в сторону первого чек-пойнта --  Ки-Вест. Плана доехать до Ки-Вест в первый же день не было. На то были две причины:
Read more

How prototype Arduino-based fan controller and break 3 boards

Image
The summer came and cooling of different systems become bigger issue than during the winter time. How do we cool down equipment normally? Just using fan. And charging stations are not excluded. Simple way is to turn the fan on when temperature reaches some threshold and turn off, when it is below. The approach will work if the system is small and the fan is silent. What if we need to control fan which creates significant noise when running at full speed? Well, try to reduce speed, when temperature only slightly above threshold, and run the full speed when it significantly above. Let's try to implement it using Arduino. Thinking naturally, we should control the voltage applied to the fan. But Arduino can not provide regulated voltage output. So we skip this approach, and start directly with PWM   PWM basics Idea of PWM conrol is that load is either fully turned on or fully off. Switching frequency is high enough and end result is indistinguishable from steady partial performance.
Read more