Dr. Brian Robert Callahan
academic, developer, with an eye towards a brighter techno-social life
Yesterday concluded a very fun journey that began back in March when I announced that I had gotten a working D compiler, GDC, the GNU D Compiler, working on OpenBSD. Yesterday, I committed a port of DMD, the Digital Mars D Compiler, to the OpenBSD ports tree.
This commit represents the first known complete DMD toolchain (dmd/druntime/phobos/tools/dub) outside of the four platforms officially supported by the D Language Foundation: FreeBSD, Linux, macOS, and Windows. There does exist a package of dmd for Haiku, but this package is of DMD version 2.067.1, released in 2015 and from a time before a D compiler was required to build DMD. Unfortunately, it is not known if this DMD compiler for Haiku ever worked, as the Pull Request mentions that there is no DRuntime or Phobos ported yet, and seems to have never been completed. There is also a package of just the DMD compiler on pkgsrc-wip but it appears to have only ever been tested on macOS.
While it is still fresh in my head, I want to write down how we got to this accomplishment. I will be giving a talk at DConf 2021 about the larger adventure of D on OpenBSD. So I won't spoil everything in this blog post, but this blog post will be a part of the larger talk.
Because you need a D compiler to build DMD, you can potentially run into the classic catch-22 of how do you build a compiler without a compiler. Fortunately, because GDC is still written in C++, pascal@ was kind enough to enable D support when he committed the gcc-11.2.0 port. And we had our starting point. I could run pkg_add gdc and get GDC. It wasn't perfect yet, but it was good enough to continue on.
Unfortunately, when you are on a platform not supported by the DLF, you quickly run into an issue building DMD where the build system attempts to download a bootstrap DMD for you. But of course, it cannot do that because it cannot find a bootstrap DMD for your platform. I ended up studying what exactly the bootstrap DMD contained by downloaded the FreeBSD bootstrap DMD. I noticed that the only things I really needed where a dmd binary, a libphobos2.a library, and a dmd.conf configuration file. The dmd binary had to come first, since I could not build phobos without it. Using the gdmd wrapper, I was able to convince dmd to build by ripping out most of the build system that handles the downloading of the bootstrap DMD from the dlang.org servers, and not caring all that much where it was that the dmd binary was being placed after being built. The D build system tries to be nice and create an ordered directory hierarchy to make packaging consistent on their end. But I didn't need that yet. I just needed the binary to exist on my machine. Anywhere would work for now.
But of course it did not build out of the box just because I had gdmd. In fact, there was a decent bit of work left before having a working DMD. What was quite fortunate was that a lot of the rote and boring work of translating OpenBSD headers into D headers had already been done. I just needed to finish up that last mile. The initial patch to complete OpenBSD support in DRuntime was quite small as a result. No patches were needed for Phobos which was quite nice.
However, I did have to fight with the build system to get things built. But it is one of those fights you only have to have once. Once I got the dmd binary and libphobos2.a library, I could create my own bootstrap DMD for OpenBSD and trick the build system into accepting it as a valid bootstrap DMD. Again, following the blueprint from the FreeBSD bootstrap DMD, I replicated the necessary directory structure and placed the binary and library in the correct location. I also placed copies of the shared libraries that the dmd binary needed, so that the bootstrap would continue to work when libraries were updated to help regenerating the bootstrap easier when that happens.
Knowing that, at least for now, I would have to host my bootstrap DMD somewhere other than the dlang.org servers, I chose my GitHub fork of DMD. That way I knew where everything was. I could now develop a port of DMD.
The funny thing about the D build system is that there are many potential ways that one could build the dmd/druntime/phobos trio. I also wanted to take as many build decisions away from the D build system as I could, prefering to have the OpenBSD ports system ultimately control the build. I had the ports system build claim it was building Phobos, since that will trigger a build of DMD and DRuntime for you automatically. I let the D build system place the resulting binaries and libraries where it felt like placing them, since I was going to override it anyway for the OpenBSD package. After the build, I used the ports system to reorganize all the files that need to be installed: dmd, libphobos2.a, and the header files and manual pages. The dmd.conf file that ships with the dmd package knows where to find all the headers, which is important since the headers are in a different location than where the D build system would put them.
Even though this package is of dmd-2.098.0, there are some additional patches for what will become dmd-2.099.0, notably ioctl handling.
But the DMD package was not ready yet. In fact, I had been sitting on a DMD package since 2.097.0 but never submitted it because there was one last bad bug to deal with: Fibers would cause an immediate segfault. I had for a long time thought it was a problem with TLS but gkoehler@ found the true culprit: the MAP_STACK flag is needed for mmap(2). In retrospect, it seems so obvious, but I am glad that we have so many talented developers in OpenBSD. As an added bonus, gkoehler@ got powerpc64 support working for GDC. He may in fact be the very first person running D on a powerpc64 machine! Those patches should be coming to DRuntime soon, but for now they are in the DMD package and will soon be in the GDC package.
With the Fibers problem solved, I was ready to submit the DMD package, which gkoehler@ ok'd and I committed.
I have already submitted ports for gdmd, dub, and the D tools. Hopefully I will be able to commit them soon.
I need to get the ioctl fixes in GDC. I am somewhat waiting for GDC to sync with DRuntime to make that job easier.
Porting LDC would be nice, but it is not high on my priority list right now. There is other D on OpenBSD work to be done. But it will happen eventually. Feel free to beat me to it!
You'll have to attend my DConf 2021 talk!
