Dr. Brian Robert Callahan
academic, developer, with an eye towards a brighter techno-social life
A few months ago, we got Oracle Developer Studio to run on and output native binaries for FreeBSD. Today, let's get another proprietary compiler going on FreeBSD: the Intel oneAPI DPC++/C++ Compiler. The latest version of this compiler as of this blog post is 2024.0.2, and this is the version we will get running on FreeBSD. This is the new Intel compiler that uses the Clang frontend from the LLVM project as its frontend, in contrast to the old version of the Intel compiler, which I believe used the Edison Design Group frontend. What makes the new Intel compiler interesting is both the Clang frontend, which should just work on FreeBSD as Clang is the built-in compiler for FreeBSD, and the backend, which is Intel's custom and proprietary backend.
As mentioned previously, a FreeBSD port of the old Intel C++ Compiler was created. It is still in the ports tree, but it only works on i386 and is version 8.1 of the compiler, released in September 2004. This update is a long time coming.
Let's get started.
Like with Oracle Developer Studio, it comes as Linux binaries; the Intel compiler is partially open source as far as I understand it. That means we will need to set up the Linuxulator. I am using FreeBSD 14.0-RELEASE, the amd64 flavor, so I am going to follow the FreeBSD Handbook, installing the CentOS 7 base and libelf packages. The commands you need are:
# sysrc linux_enable="YES" # service linux start # pkg install linux_base-c7 linux-c7-elfutils-libelf
And that's it. The CentOS 7 libraries, though old, are enough for the Intel compiler to work.
First, I installed some tools that we will need. The installer script needs the GNU df, tar, and bash. We will need to install these:
# pkg install bash coreutils gtar
I then needed to make sure GNU df is found first and gtar is also found first, and that they are named df and tar. I did that by running:
$ mkdir ~/bin $ cd ~/bin $ cp /usr/local/bin/gdf df $ cp /usr/local/bin/gtar tar $ export PATH=~/bin:$PATH
Now the installer will not complain.
Let's download and run the installer now. I want to install the compiler to /opt/intel so I will make sure to run the script as root:
$ cd $ fetch https://registrationcenter-download.intel.com/akdlm/IRC_NAS/bb99984f-370f-413d-bbec-38928d2458f2/l_dpcpp-cpp-compiler_p_2024.0.2.29_offline.sh $ sudo bash l_dpcpp-cpp-compiler_p_2024.0.2.29_offline.sh -a -s --eula accept --ignore-errors
...but eventually I gave up; while the installation completes, because the installer cannot find G++ it does not install all the programs, most notably the icx and icpx compilers. This is a failing on Intel's part: I have to imagine the reason the installer is looking for G++ is because it thinks without it, you don't have the necessary headers and libraries. But this is untrue on FreeBSD as those things come installed by default. Installing the gcc package does not fix things, so Intel should improve their installation program to fix things. Even so, there is a --ignore-errors flag, and I think that flag should ignore if G++ is missing and simply install everything anyway.
There is a workaround: I installed the compiler using the same command on my WSL environment on Windows. I was then able to bundle the resulting /opt directory into a tarball and move and extract the tarball on the FreeBSD machine. This worked fine, and now I have the compiler installed to /opt/intel. I also removed the ~/bin directory as that was only needed for installation and we didn't install it from the provided installer after all.
If I run /opt/intel/oneapi/compiler/latest/bin/icx --version, I get back:
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/intel/oneapi/compiler/2024.0/bin/compiler Configuration file: /opt/intel/oneapi/compiler/2024.0/bin/compiler/../icx.cfg
This definitely looks like Clang. For comparison, if I run cc --version, I get back:
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) Target: x86_64-unknown-freebsd14.0 Thread model: posix InstalledDir: /usr/bin
This makes me wonder something: if Intel didn't do all that much modification, could we convince icx that it is targeting FreeBSD? On Clang, we could do this with the -target flag.
Indeed, if I run icx -target x86_64-unknown-freebsd14.0 --version, I get back this:
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213) Target: x86_64-unknown-freebsd14.0 Thread model: posix InstalledDir: /opt/intel/oneapi/compiler/2024.0/bin/compiler Configuration file: /opt/intel/oneapi/compiler/2024.0/bin/compiler/../icx.cfg
Yup, look at that. Now icx believes it is targeting FreeBSD. This somewhat indicates that Intel could very simply and trivially provide binaries for at least FreeBSD, but probably all the BSDs, if they wanted to.
Let's try to compile this hello world program:
#include <stdio.h>
int main(void) { puts("Hello"); return 0; }
If we compile it with /opt/intel/oneapi/compiler/latest/bin/icx -v hello.c, we see:
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/intel/oneapi/compiler/2024.0/bin/compiler Configuration file: /opt/intel/oneapi/compiler/2024.0/bin/compiler/../icx.cfg "/opt/intel/oneapi/compiler/2024.0/bin/compiler/clang" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -dumpdir a- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name hello.c -mrelocation-model static -fveclib=SVML -mframe-pointer=none -menable-no-infs -menable-no-nans -fapprox-func -funsafe-math-optimizations -fno-signed-zeros -mreassociate -freciprocal-math -fdenormal-fp-math=preserve-sign,preserve-sign -ffp-contract=fast -fno-rounding-math -ffast-math -ffinite-math-only -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -mllvm -x86-enable-unaligned-vector-move=true -tune-cpu generic -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/brian -resource-dir /opt/intel/oneapi/compiler/2024.0/lib/clang/17 -internal-isystem /opt/intel/oneapi/compiler/2024.0/bin/compiler/../../opt/compiler/include -internal-isystem /opt/intel/oneapi/compiler/2024.0/lib/clang/17/include -internal-isystem /usr/local/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -fdebug-compilation-dir=/home/brian -ferror-limit 19 -fheinous-gnu-extensions -fgnuc-version=4.2.1 -fcolor-diagnostics -vectorize-loops -vectorize-slp -D__GCC_HAVE_DWARF2_CFI_ASM=1 -fintel-compatibility -fintel-libirc-allowed -mllvm -disable-hir-generate-mkl-call -mllvm -loopopt=1 -floopopt-pipeline=light -mllvm -intel-abi-compatible=true -o /tmp/icx-7df38f729d/hello-6860a3.o -x c hello.c clang -cc1 version 17.0.0 based upon LLVM 17.0.0git default target x86_64-unknown-linux-gnu ignoring nonexistent directory "/include" #include "..." search starts here: #include <...> search starts here: /opt/intel/oneapi/compiler/2024.0/bin/compiler/../../opt/compiler/include /opt/intel/oneapi/compiler/2024.0/lib/clang/17/include /usr/local/include /usr/include End of search list. "/usr/bin/ld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/crt1.o /usr/lib/crti.o /usr/lib/crtbegin.o -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/compiler/2024.0/lib/clang/17/lib/x86_64-unknown-linux-gnu -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../lib -L/opt/intel/oneapi/compiler/2024.0/bin/compiler/../../opt/compiler/lib -L/lib -L/usr/lib /tmp/icx-7df38f729d/hello-6860a3.o -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -Bstatic -limf -Bdynamic -lm -lgcc --as-needed -lgcc_s --no-as-needed -Bstatic -lirc -Bdynamic -ldl -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed -Bstatic -lirc_s -Bdynamic /usr/lib/crtend.o /usr/lib/crtn.o
That's close. The compilation might be fine, but the linking is definitely not fine. Fortunately, we know how to fix it.
Similarly to Oracle Developer Studio, the Intel compilers need a small wrapper program to help them along. You could get away with a shell script, but I will use D to write the wrapper programs. Unlike Oracle Developer Studio, we don't need to manipulate the preprocessor defines as the -target flag does that for us. Changing the target also changes the linker invocation to be correct. The wrapper program looks like this:
import std.process;
int main(string[] args) {
string[] av;
av ~= "/opt/intel/oneapi/compiler/latest/bin/icx";
av ~= "-target";
version (FreeBSD_14) av ~= "x86_64-unknown-freebsd14.0";
else version (FreeBSD_13) av ~= "x86_64-unknown-freebsd13.0";
av ~= "-fno-builtin";
foreach (i; 1 .. args.length)
av ~= args[i];
return spawnProcess(av).wait;
}
Save it as icx.d and then all we need to do is compile it with LDC and we'll be good:
$ ldc2 -O icx.d
That will create a binary named icx. I put it in /usr/bin but you could put it in /usr/local/bin if you were going to make this into a port or something.
Similar needs to be done for the icpx wrapper:
import std.process;
int main(string[] args) {
string[] av;
av ~= "/opt/intel/oneapi/compiler/latest/bin/icpx";
av ~= "-target";
version (FreeBSD_14) av ~= "x86_64-unknown-freebsd14.0";
else version (FreeBSD_13) av ~= "x86_64-unknown-freebsd13.0";
av ~= "-fno-builtin";
foreach (i; 1 .. args.length)
av ~= args[i];
return spawnProcess(av).wait;
}
Save it as icpx.d and run:
$ ldc2 -O icpx.d
And you will have yourself an icpx binary.
-fno-builtin?When you change the target icx is compiling for, it greatly changes the linker invocation. This actually makes sense, as the Linux libraries actually depend on glibc-specific features. For example, anything more than trivial programs will give you linker errors saying you are missing functions such as __vsnprintf_chk. This is a bit unfortunate; when I created a library with those functions and linked it in, everything worked fine. So again, Intel could provide a fully working compiler and support libraries for all the BSDs if they so choose to. To circumvent the need to use Intel's support libraries, you add -fno-builtin.
As a test, I installed the FreeBSD 14.0-RELEASE source code on my machine and built a kernel with:
# cd /usr/src # CC=icx COMPILER_TYPE=clang make buildkernel
And then I waited around for a little bit, but it did in fact complete the build. When I run readelf -x .comment /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel, I see the comment I am expecting: Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231223). The icx built kernel is about 3% larger than the kernel that came installed on the system.
I then ran make installkernel and once confirming my new kernel was in fact installed, I rebooted the machine. To my absolute astonishment, it booted just fine and put me into my X environment.
In a hilarious twist, the only thing that appears to have been miscompiled is the Linuxulator, as I get a bunch of these messages in my dmesg:
kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linux.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linux.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linux64.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linux64.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 108 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/fdescfs.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linprocfs.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linprocfs.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linsysfs.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linsysfs.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linprocfs.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linprocfs.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 310 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/linux_common.ko - unsupported file type KLD linsysfs.ko: depends on linux_common - not available or version mismatch linker_load_file: /boot/kernel/linsysfs.ko - unsupported file type kldload: unexpected relocation type 42, symbol index 108 link_elf_obj: symbol __stack_chk_guard undefined linker_load_file: /boot/kernel/fdescfs.ko - unsupported file type
Of course, there could be more things that are miscompiled, but only the Linuxulator is what triggered as bad on my system.
I did not test the C++ compiler as heavily, but everything I did throw at it compiled without issue. I would be surprised if icpx would fail to compile anything clang++ could compile.
The Intel oneAPI DPC++/C++ will run just fine on FreeBSD. It is certainly production-quality for userland programs. It is nearly acceptable as a kernel compiler; it might in fact already be acceptable, the problem could be in my wrapper program. It could be immediately improved by Intel actually supporting FreeBSD with the support library and native compiler binaries; this is actually quite easy, but Intel may lack the will to do so.
In any event, we can definitely add the Intel compiler to compilers that can output native binaries for FreeBSD, and likely all the BSDs. I think that's neat.
