Dr. Brian Robert Callahan
academic, developer, with an eye towards a brighter techno-social life
Yes, I am aware that I am interrupting our self-hosting PL/0 compiler series but I think it will be worth it. Earlier today, I got the J language system running on OpenBSD. I find it important to write these up, because it helps me preserve my own knowledge of what I did and hopefully it will help others porting languages to their favorite *BSD.
J is an array programming language. That is to say, the fundamental data type in J is the array. J is a successor language of APL. I have a little bit of experience with APL by way of GNU APL. Back in 2017, I imported GNU APL into the ports tree along with a font for use with APL, as APL uses lots of symbols not available on the standard ASCII keyboard. On the same day, in fact only three minutes after importing GNU APL into the ports tree, I imported kona into the ports tree. Kona is an open source reimplementation of K, another array programming language heavily influenced by APL. Unlike APL, K uses only the ASCII character set, making K a lot easier to program with the keyboards I have.
There was a third array programming language from the APL family tree that I discovered and was interested in: J. I tried porting J back at that time but failed because it had a very complex build system. In comparison, GNU APL was a fairly straightforward port, as it used the GNU Autoconf system to configure GNU APL for OpenBSD and the build just worked after that. Kona was even easier: it's all written in simple C and comes with a simple GNU Makefile, no configuration needed.
I put the failed attempt to port J into the openbsd-wip repository and all but forgot about it.
I came across a YouTube video that was a review of solving a single programming challenge with sixteen different languages. J was among the list (the video maker is apparently a big fan of array programming languages) and it reminded me that I should probably give porting J another try. And I'm glad I did. In the interim years, either I have gotten better at porting software to OpenBSD or the J build system got a lot more platform-independent and platform-agnostic. Probably a combination of both. In any event, I grabbed a copy of my original J porting attempt and updated the version numbers and other essentials, and got to work.
One of the first things I check when making a new port is seeing what the build system is. Most of the time, you have a commonly used and understood system, such as plain ol' make, GNU autoconf/automake, CMake, meson, or SCons. Whatever your feelings about these build systems are, the reality is the fact that they are well understood means that the OpenBSD ports system can plug right into them and, if upstream has taken care to write good build files, you can often get things working with minimal headache.
J did not have such a system back when I first attempted the port and it does not have one today. What J has today is a combination of shell scripts that control GNU Makefiles. OK, that's not so bad. They're even POSIX shell scripts, no hidden GNU Bash dependencies. Thank you J team!
But it does mean that we will need to write a custom do-build routine for this port, since there is nothing that the ports system natively understands and therefore the ports system cannot abstract the work away.
Reading through some of the documentation that came with the source code, I discovered that if you'd like to build the entire J system, you can do so by executing the build_all.sh script found in the make2 directory of the J source code. That's our clue for what our do-build routine should be:
do-build:
cd ${WRKSRC}/make2 && ./build_all.sh
Except that didn't work as hoped. The build errored out pretty quickly, informing me that it didn't know what operating system this is, and apparently it's not adventurous enough to try out OpenBSD and seeing what happens. Alright fine, let's go find where it figues out what operating system this is. We're just going to lie and say we're Linux. The magic location is in the file jsrc/js.h there is an #ifdef __linux__ line that I changed to #ifdef __OpenBSD__. Then I restarted the build.
Much to my amazement, the build ran to completion at this point. There is no way things could be that easy. It wasn't that easy, but it was pretty close.
One thing that I noticed that we tend to not like in OpenBSD ports is that -Werror was in every compiler invocation line. That had to go. Additionally, it was hardcoding -O2 as an optimization flag. I instead changed it to take whatever CFLAGS the user has in their environment, which is the common solution to hardcoded optimization flags in OpenBSD ports.
Additionally, I noticed that J wanted to use flags that bring in Intel AVX extensions. But if we use those, then the package we create will not be usable on all amd64 systems as it seems that there is no runtime detection for CPU extensions, only build-time options. Fortunately, the build system has a mechanism to disable the use of these extensions. Unfortunately, J still requires SSE3 instructions, so there are still a small handful of amd64 machines that will not be able to use J, but that will only be a problem for the original amd64 CPUs.
I may be wrong about J not having any runtime detection. But the documentation is not clear on the matter and the naming convention makes it even less clear, so if you know let me know and I'll update the port accordingly.
Now we need to install J and test it out.
J is very particular with where its standard library lives in relation to the jconsole binary. The jconsole binary is also very particular as to where its shared libraries are location in relation to it. Unfortunately, this information appears to be hardcoded and untangling it I found to be too difficult. So I chose the next best option: I installed the J system into share/j and wrote a shell script that will cd into the correct directory and launch jconsole from there. It does mean that if you want to give jconsole a program as an argument, you need to provide the full path to the program. I can live with that. I think we can all live with that, since you would need to do that anyway because of how J lives together, so nothing has been lost.
Let's try to run it.
I installed the package and tried to run jconsole. It complained that it could not execute cat /proc/cpuinfo. That makes sense, since OpenBSD does not support procfs. Not a problem, I removed the call from jlibrary/system/main/stdlib.ijs, which is where that particular call lives. A rebuild and reinstall later, and I had a jconsole that was behaving correctly and not giving me a spurious procfs error.
Now it was time to run the testing apparatus.
J comes with a battery of tests, though you could be forgiven for missing how to run them. You cannot run the J test programs directly. You have to run them via a helper J test monitor program. The information is in make2/make.txt and I missed it the first time. All the tests run successfully except gss.ijs, which seems to run fine until the very end and then you get an Abort Trap. I will have to figure that out. But all other tests run successfully. That's a great sign. At this point, I'm confident that the port is ready to be sent to ports@.
I had lots of tendrils left over from my initial attempt at porting J. At this point, they all had to go because I had a finished port. I also needed to set ONLY_FOR_ARCHS=amd64. Not because J only works on amd64 but because I know based on directory structures that there will need to be extra work to get J running on i386, armv7, and arm64. I'd prefer to work on that in-tree because there's no reason for amd64 users to have to wait.
If you'd like to try the port (and it has not yet been committed), you can do so here.
I was very happy with the porting effort. J turned out to be a moderately difficult port because of its interesting build system and need to disable AVX extensions, but it was made a bit easier by being quick to compile, so recompiling due to a header change and a runtime fix was not time consuming. I was also happy that J is quite operating system-agnostic, so long as you're willing to lie. Perhaps the J people will consider renaming Linux to Unix, since it seems there is little to nothing actually Linux-specific. And doing so may help encourage others to port and package J. In any event, thanks to the J team for making the build straightfoward if a little unconventional and for making a useful and interesting programming language.
If you're interested in array programming languages, or just want to try out something new, give J a spin.
