Dr. Brian Robert Callahan
academic, developer, with an eye towards a brighter techno-social life
I have a mini PC that I bought back in May. It's an OK machine for the price. It's a little more OK of a machine for me since I got it for about $100. At the moment it runs FreeBSD 14.2 and acts as a home NAS. It was cheap, it is low power, and with a 1 TB Samsung T7 SSD it meets my needs. Honestly, it is my favorite type of hardware: something that gets out of my way and lets me do the things I want to do. Incidentally, this is also why I like FreeBSD and ZFS for running a NAS.
Of course, with an Intel N100 CPU and 12 GB of RAM, the machine does not really do all that much for its specs. Even with ZFS and Samba running, I'm not using anywhere near all the machine that I have.
I was thinking that I want to make a massively cross compilation setup on the NAS. I suppose I could do this on my new MacBook Pro, and it would probably be faster on the Mac, but the Mac does enough work and the NAS does not do enough work.
When I make a release of oksh, I have to fire up a whole lot of virtual machines so that I can build test on different operating systems. And there really are only a few operating systems that get build tested: FreeBSD, NetBSD, macOS, and glibc-based Linux. OpenBSD does not need to be tested. And everything else can be fixed when issues arise, if they ever arise (is anyone actually using Unixware 7 and using oksh on it?).
And let's be real, when it comes to run-time testing, the only platform that gets any real run-time testing is FreeBSD. I suppose macOS will now too, but seeing as I simply add portability goo to the in-base OpenBSD ksh and do not modify it, run-time testing is of limited importance to me.
If I can set up macOS cross compilation, NetBSD and Linux should be easy to follow suit.
At the end of this, I will be able to produce native macOS/aarch64 binaries from C, C++, and D code.
LLVM has a tutorial on how to set up a cross compilation environment here, but I think having a more concrete example would be more beneficial, hence this blog post.
I needed to first make a tarball of the macOS headers and libraries. Fortunately, these are all conveniently found in one place: /Library/Developer/CommandLineTools/SDKs.
All we need to do is tar that up and send it to the NAS.
I made a new ZFS dataset with sudo zfs create -o mountpoint=/Library zroot/Library. I then extract the tarball that was sent over so that the path to the macOS headers and libraries on the FreeBSD machine is exactly the same as on the MacBook Pro.
Clang, from the LLVM project, is by default built as a massively cross compilation system. That is to say, unless you specify options to do otherwise, clang can output code to any target machine that LLVM understands. All you need to do is set the appropriate -target flag. For example, if we want to target macOS/aarch64, all we need to do is give clang the -target aarch64-apple-darwin24.2.0 flag and clang will compile for macOS/aarch64 running Sequoia 15.2. But as we will soon see, that is a start but it is not enough.
On FreeBSD, the in-base clang compiler is specially modified to only support the host machine. But that's no bother; we can simply install any version of LLVM from ports. As of this writing, LLVM 19 is the latest major release, so I will install that with sudo pkg install llvm19. This will give me a clang19 and clang++19 compiler that can target macOS/aarch64 (and everything else LLVM knows about).
Traditionally, there are three primary steps for languages like C to go from source code to binaries: compiling, assembling, and linking. If you ever look under the hood at something like GCC, you'll notice that there is a compiler program cc1 that compiles C source code into assembly, an assembler as that assembles the assembly into object code, and a linker ld that combines all that object code and libraries together to produce a working binary.
Clang combines the compiling and assembling together into one step. Perhaps it is more correct to say that clang is able to directly output object code from source code. It's a little more complicated than that, but that's outside the scope of what we want to accomplish today.
What we care about is that we need to do two things:
If we can do that, we can have our complete cross compiler.
Let's start with C, though C++ will be extremely similar. We know we need the -target aarch64-apple-darwin24.2.0 to ensure we generate code for macOS/aarch64.
What else do we need? We need to make sure that clang, when compiling for macOS, finds the macOS headers and not the FreeBSD headers.
There is a flag for that: --sysroot. The flag takes an argument, the path to where your headers and libraries live. For us, that takes the form of --sysroot /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk which will do the right thing, as there are /usr/include and /usr/lib directories in there, and that is what clang wants to see.
If I write a simple C file:
#include <stdio.h>
int
main(void)
{
puts("Hello world");
return 0;
}
Let's see what happens if we try to compile it to an object file:
clang19 -target aarch64-apple-darwin24.2.0 --sysroot /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk -O2 -c hello.c
Hey, that worked. So that's what we need to compile C code. It is exactly the same for C++ code; all you have to do is swap clang19 with clang++19.
Indeed, if we run file hello.o we get back hello.o: Mach-O 64-bit arm64 object, flags:<|SUBSECTIONS_VIA_SYMBOLS>.
Just like the in-base clang only supports the native system, the in-base linker ld also only supports the native system. Fortunately, our llvm19 package also installed another linker, lld19.
Linking exposes an interesting problem in a way that compiling and assembling does not: the problem of different binary formats. Of the major operating systems, there are three main binary formats. Windows uses PE, the BSDs and Linux use ELF, and macOS uses Mach-O.
Fortunately, clang knows how to write object files in all three formats. But that's the easy part. The difficult part is combining many object files with libraries with other support code to create binaries in each format; that is the job of the linker.
Because we are on FreeBSD, our in-base linker only understands ELF. The linker from the llvm19 package does understand Mach-O, but needs to be used as ld64.lld19. We can use the -fuse-ld=/usr/local/bin/ld64.lld19 flag with clang to ensure we use the correct linker.
The linker is called the way it is because ld64 is the name of Apple's Mach-O linker. The LLVM linker, lld, is actually three (four) separate linkers: ELF, PE, Mach-O (and WebAssembly) all smashed into one executable. They don't share all too much code. But they are all one big executable and the point here is we can't call it by the name lld; the linker actually checks to see how it was called to know what kind of object files it will deal with. So for us to target macOS, we will call it as ld64.lld.
In theory seeing as ld64 is open source, I could compile it on the FreeBSD machine and use that, but I have not found any issues using lld.
Finally, we need to add the two required arguments to ld64.lld19: -arch arm64e and -platform_version macos 15.0.0 15.2.
Because we need to pass these flags from clang to lld, we have to write them in the slightly awkward -Wl,-arch,arm64e and -Wl,-platform_version,macos,15.0.0,15.2 forms.
But once we do that, we have done it. All together, we can run:
clang19 -O2 -target aarch64-apple-darwin24.2.0 --sysroot /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk -fuse-ld=/usr/local/bin/ld64.lld19 -Wl,-arch,arm64e -Wl,-platform_version,macos,15.0.0,15.2 -o hello hello.c
And we get a hello executable that will run on macOS. Running file hello produces hello: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>.
I copied the hello executable over to my MacBook Pro and it ran without issues. I repeated the process with a C++ hello world file and it worked perfectly too.
I downloaded the FreeBSD version of LDC from their GitHub repository. As it's LLVM under the hood, it understands all the targets LLVM understands.
I hit a bit of a snag with the version of LDC I downloaded: the LDC team claims they built the latest version of LDC with LLVM 19 but it is very obvious they built it with LLVM 15 on FreeBSD. That seems bad seeing as the in-base version of clang on FreeBSD 14.2 is 18.1.6. I ended up rebuilding LDC so that it was built with LLVM 19, though I still used the phobos and druntime libraries I downloaded.
LDC has very different flags compared to clang. To save a bit of time, the LDC version of -target is --mtriple=aarch64-apple-darwin24.2.0, which is the only compiler flag you need. The linker flags were more difficult to figure out. Assuming you downloaded LDC from GitHub to ~/ldc2-1.40.0-osx-arm64, the linker flags you need are:
--gcc /usr/local/llvm19/bin/clang -Xcc --sysroot -Xcc /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk -Xcc -fuse-ld=/usr/local/bin/ld64.lld19 -Xcc -Wl,-arch,arm64e -Xcc -Wl,-platform_version,macos,15.0.0,15.2 -L-L`echo $HOME`/ldc2-1.40.0-osx-arm64/lib
But with that, I was able to cross compile a hello world program in D that worked correctly when I copied it over to the MacBook Pro.
All we have to do is repeat this process for any other operating systems we want to have a cross compilation environment for, and we'll be good. Yes, it will get more complicated as you have more complex projects with more headers and libraries, but as a start for simple console-based C, C++, and D programs, this will fit the bill just fine.
