Dr. Brian Robert Callahan
academic, developer, with an eye towards a brighter techno-social life
Recently, I taught the mold linker how to find shared libraries on OpenBSD. This was the last puzzle piece needed to get mold working on OpenBSD. Testing on some simple applications, like oksh, produced working executables.
I would like to go a bit further and push mold to its limits. I want to know what would happen if mold was the only linker on our system.
For those that don't know, mold (short for Modern Linker) is a Unix linker developed by Rui Ueyama, who also designed lld, the LLVM linker, and chibicc, which we looked at some time ago. As of version 2.0.0, it is licensed under the MIT license, though the required oneTBB library is Apache 2.0 licensed. Mold is billed as being the fastest linker on the market, heavily using modern techniques such as better algorithms and parallelization. Mold is always built as a cross linker supporting many different targets. Mold is certainly a welcome addition to the rather small number of open source Unix linkers: GNU ld and LLVM lld on all Unix systems, and the Sun linker on Solaris and its derivatives. There is also GNU gold, though Gentoo claims that it is not very active, and I haven't heard of many people using it.
It's quite straightforward. There are only a few items that mold needed to be taught:
-run and Link-Time Optimization; there is no way to do this programmatically on OpenBSD, but we can hardcode the path.-z wxneeded and -z nobtcfi flags; there is code out there that needs the help.-pie and --execute-only.You can see all the changes I had to make here. After that, just use CMake and Ninja, and you've got yourself a mold executable.
I moved /usr/bin/ld.lld and /usr/bin/ld.bfd out of my PATH so that I could be 100% certain only mold was being used. I installed everything to /usr, moved /usr/bin/mold to /usr/bin/ld.mold and hardlinked /usr/bin/ld to /usr/bin/ld.mold. I then built mold another two times and repeated this process, so that I could be sure that a mold linked with mold could link mold. Now I was ready to build all of OpenBSD.
You can follow the directions in release(8) to build the kernel. Unfortunately, this did not work. The linker complained that it could not determine the file type for the linker script being used. This is a quirk in how mold determines if something is a linker script. Mold checks if a file is at least four bytes in size and that the first four bytes are printable using isprint(3). The isprint(3) function checks if its argument is an ASCII character in the range 0x20 to 0x7e, inclusive. The first character in the linker script the OpenBSD kernel uses is a newline, which is 0x0a. This causes mold's linker script detection logic to fail. The simple fix is to teach mold that newlines are acceptable characters when determining if something is a linker script.
With this fix, mold reported a new problem: it did not understand the PHDRS token. Mold only understands a very limited subset of the linker command language, just enough for linking with glibc. That might be good enough for glibc, but it is not good enough for the OpenBSD kernel.
This is the gap.link script the OpenBSD/amd64 kernel uses:
PHDRS {
text PT_LOAD FILEHDR PHDRS;
rodata PT_LOAD;
data PT_LOAD;
bss PT_LOAD;
}
SECTIONS {
.text : ALIGN(4096) {
LONG(0xcccccccc);
. += 964;
. = ALIGN(4096);
endboot = .;
PROVIDE (endboot = .);
. = ALIGN(4096);
. += 1444;
. = ALIGN(16);
*(.text .text.*)
} :text =0xcccccccc
.rodata : {
LONG(0xcccccccc);
. += 2791;
. = ALIGN(16);
*(.rodata .rodata.*)
} :rodata =0xcccccccc
.data : {
LONG(0xcccccccc);
. = . + 3052; /* fragment of page */
. = ALIGN(16);
*(.data .data.*)
} :data =0xcccccccc
.bss : {
. = . + 2315; /* fragment of page */
. = ALIGN(16);
*(.bss .bss.*)
} :bss
}
Hopefully this gets implemented soon; otherwise I'll probably have to hire a graduate student to implement it for me.
Just because building the kernel failed doesn't mean there isn't a lot left to try out. There is still the whole OpenBSD userland. Here again, we can use the release(8) manual page for instructions to build userland. I also unhooked lld from the build to make sure there was no accidental usage of another linker.
Building userland initially appeared much more successful than the kernel. There were still some failures along they way.
The first failure was libcrypto, which failed with these messages:
mold: error: sha1-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0x7 against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: aes-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0xfa4 against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: aesni-sha1-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0x7 against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: aesni-sha1-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0xe against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: sha1-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0xe against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: rc4-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0x39 against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC mold: error: rc4-x86_64.so:(.text): R_X86_64_PC32 relocation at offset 0x690 against symbol `OPENSSL_ia32cap_P' can not be used; recompile with -fPIC
I admit I am not sure why this fails; it works with both ld.lld and ld.bfd.
The second failure was ld.so, like the kernel because of missing linker script support:
mold: fatal: /usr/src/libexec/ld.so/amd64/ld.script:1: PHDRS
^ unknown linker script token
The linker script in question looks like this:
PHDRS
{
rodata PT_LOAD FILEHDR PHDRS FLAGS (4);
text PT_LOAD FLAGS (1);
btext PT_LOAD FLAGS (0x08000005);
data PT_LOAD;
random PT_OPENBSD_RANDOMIZE;
relro PT_GNU_RELRO;
dynamic PT_DYNAMIC;
note PT_NOTE;
}
SECTIONS
{
. = 0 + SIZEOF_HEADERS;
/* RODATA */
.gnu.hash : { *(.gnu.hash) } :rodata
.dynsym : { *(.dynsym) } :rodata
.dynstr : { *(.dynstr) } :rodata
.rodata : { *(.rodata .rodata.*) } :rodata
.eh_frame : { *(.eh_frame) } :rodata
/* TEXT */
. = ALIGN(0x1000);
.boot.text :
{
. = ALIGN(0x1000);
boot_text_start = .;
*(.boot.text)
. = ALIGN(0x1000);
boot_text_end = .;
} :btext =0xcccccccc
.text : { *(.text .text.*) } :text =0xcccccccc
/* RELRO DATA */
. = DATA_SEGMENT_ALIGN (0x100000, 0x1000);
.openbsd.randomdata :
{
*(.openbsd.randomdata .openbsd.randomdata.*)
} :data :relro :random
.data.rel.ro : { *(.data.rel.ro.local*) *(.data.rel.ro*) } :data :relro
.dynamic : { *(.dynamic) } :data :relro :dynamic
.got : { *(.got.plt) *(.got) } :data :relro
. = DATA_SEGMENT_RELRO_END (0, .);
/* BOOTDATA */
. = ALIGN(0x1000);
boot_data_start = .;
.rela.dyn :
{
*(.rela.text .rela.text.*)
*(.rela.rodata .rela.rodata.*)
*(.rela.data .rela.data.*)
*(.rela.got)
*(.rela.bss .rela.bss.*)
} :data
/* XXX .rela.plt is unused but cannot delete: ld.bfd zeros DT_RELASZ then! */
.rela.plt : { *(.rela.plt) } :data
.note : { *(.note.openbsd.*) } :data :note
.hash : { *(.hash) } :data
.boot.data : { *(.boot.data .boot.data.*) } :data
boot_data_end = .;
/* DATA */
. = ALIGN(0x1000);
.data : { *(.data .data.*) } :data
.bss : { *(.dynbss) *(.bss .bss.*) *(COMMON) } :data
. = DATA_SEGMENT_END (.);
/DISCARD/ : { *(.note.GNU-stack) }
}
There were a number of failures in the sys/arch/amd64/stand directory:
sys/arch/amd64/stand/biosboot:
mold: fatal: /usr/src/sys/arch/amd64/stand/biosboot/ld.script:1: PHDRS
^ unknown linker script token
sys/arch/amd64/stand/boot:
ld -nostdlib -Bstatic -Ttext 0x40120 -N -x -nopie -znorelro -melf_i386 -L/usr/libdata -o boot.new srt0.o conf.o boot.o bootarg.o cmd.o vars.o gidt.o mdrandom.o run_amd64.o cmd_i386.o dev_i386.o exec_i386.o gateA20.o machdep.o bioscons.o biosdev.o diskprobe.o memprobe.o time.o softraid_amd64.o alloc.o ctime.o exit.o getchar.o hexdump.o memcmp.o memcpy.o memmove.o memset.o printf.o putchar.o snprintf.o strcmp.o strerror.o strlen.o strncmp.o strncpy.o strtol.o strtoll.o close.o closeall.o cons.o cread.o dev.o disklabel.o dkcksum.o fchmod.o fstat.o lseek.o open.o read.o readdir.o stat.o elf32.o elf64.o loadfile.o arc4.o ufs.o ufs2.o aes_xts.o bcrypt_pbkdf.o blowfish.o explicit_bzero.o hmac_sha1.o pkcs5_pbkdf2.o rijndael.o sha1.o sha2.o softraid.o ashldi3.o ashrdi3.o divdi3.o lshrdi3.o moddi3.o qdivrem.o strlcpy.o adler32.o crc32.o inflate.o inftrees.o
.text has incorrect file offset 0x6120 (should be 0x120)
.data has incorrect file offset 0x168fc (should be 0x128fc)
sys/arch/amd64/stand/fdboot:
ld -nostdlib -Bstatic -Ttext 0x40120 -N -x -nopie -znorelro -melf_i386 -L/usr/libdata -o fdboot.new srt0.o conf.o boot.o bootarg.o cmd.o vars.o gidt.o mdrandom.o run_amd64.o cmd_i386.o dev_i386.o exec_i386.o gateA20.o machdep.o bioscons.o biosdev.o diskprobe.o memprobe.o time.o alloc.o ctime.o exit.o getchar.o hexdump.o memcmp.o memcpy.o memmove.o memset.o printf.o putchar.o snprintf.o strcmp.o strerror.o strlen.o strncmp.o strncpy.o strtol.o strtoll.o close.o closeall.o cons.o cread.o dev.o disklabel.o dkcksum.o fchmod.o fstat.o lseek.o open.o read.o readdir.o stat.o elf32.o elf64.o loadfile.o arc4.o ufs.o ashldi3.o ashrdi3.o divdi3.o lshrdi3.o moddi3.o qdivrem.o strlcpy.o adler32.o crc32.o inflate.o inftrees.o
.text has incorrect file offset 0x2120 (should be 0x120)
sys/arch/amd64/stand/pxeboot:
ld -nostdlib -Bstatic -Ttext 0x40120 -N -x -nopie -znorelro -melf_i386 -L/usr/libdata -o pxeboot srt0.o conf.o devopen.o open.o machdep.o exec_i386.o cmd_i386.o run_amd64.o gidt.o mdrandom.o biosdev.o bioscons.o gateA20.o memprobe.o diskprobe.o time.o pxe.o pxe_call.o pxe_net.o pxe_udp.o softraid_amd64.o boot.o
cmd.o vars.o bootarg.o alloc.o exit.o getchar.o getfile.o getln.o globals.o hexdump.o strcmp.o strlen.o strncmp.o memcmp.o memcpy.o memmove.o memset.o printf.o putchar.o snprintf.o strerror.o strncpy.o strtol.o strtoll.o ctime.o strlcpy.o strlcat.o aes_xts.o bcrypt_pbkdf.o blowfish.o explicit_bzero.o hmac_sha1.o
pkcs5_pbkdf2.o rijndael.o sha1.o sha2.o softraid.o close.o closeall.o dev.o disklabel.o dkcksum.o fchmod.o fstat.o ioctl.o lseek.o read.o stat.o write.o cread.o readdir.o cons.o loadfile.o arc4.o elf32.o elf64.o ether.o net.o netif.o rpc.o bootp.o bootparam.o ufs.o ufs2.o nfs.o tftp.o ashldi3.o ashrdi3.o divdi3.o
lshrdi3.o moddi3.o qdivrem.o udivdi3.o umoddi3.o adler32.o crc32.o inflate.o inftrees.o
mold: error: pxe_call.o:(.text): relocation R_386_PC16 against real_to_prot out of range: 65650 is not in [-32768, 32768)
mold: error: pxe_call.o:(.text): relocation R_386_PC16 against real_to_prot out of range: 65608 is not in [-32768, 32768)
sys/arch/amd64/stand/efiboot/bootx64:
mold: fatal: /usr/src/sys/arch/amd64/stand/efiboot/bootx64/../ldscript.amd64:3: OUTPUT_ARCH(i386:x86-64)
^ unknown linker script token
After that, I noticed that new C runtime files were installed. Because of this, we experienced more problems.
It appears that mold mislinks C runtime files such as crtbegin.o. Even trying to link a basic true.c file that just returns 0 gives the following error:
mold: error: /usr/lib/crtbegin.o:(.text): R_X86_64_PC32 relocation at offset 0x7a against symbol `' can not be used; recompile with -fno-PIC
This is obviously untenable; the system can't be used like this.
I did not attempt to build Xenocara or ports.
It is certainly possible that these problems are my fault; OpenBSD has made a large number of local changes to lld to ensure it works correctly for OpenBSD's needs, and it is more than possible that I have not attended to them all. With that said, there are currently too many problems to use mold as the system linker on OpenBSD. Even so, it is possible to use mold as a boutique linker for userland programs on OpenBSD, and that's a good start.
Running doas sysupgrade -sf gave me my original system back.
