Brian Robert Callahan

[prev]

[next]

2021-08-17
Let's write a compiler, part 4: Testing

All source code for this blog post can be found here.

Today, let's write some infrastructure to make our lives easier when it comes to testing that our compiler works correctly. We will write a Makefile and a testing apparatus for our PL/0 compiler.

Makefile

Since our PL/0 compiler is a single file C application, we have gotten away with just running the C compiler directly on our source code. Even so, make is fewer characters than cc pl0.c so a Makefile would be nice for that reason alone. It need not be fancy, it just has to work. Let's write the bare minimum to compile and clean up after ourselves:

# pl0c Makefile

CC =		cc
CFLAGS =	-g -O2 -DHAVE_STRTONUM

PROG =	pl0c
OBJS =	pl0c.o strtonum.o

all: ${OBJS}
	${CC} ${LDFLAGS} -o ${PROG} ${OBJS}

test:
	cd tests && ./test.sh

clean:
	rm -f ${PROG} ${OBJS} ${PROG}.core

Done. Now we can run make to recompile our PL/0 compiler. And we can clean up after ourselves with make clean.

Adding a testing apparatus

Automated testing is great. I would like to be able to run make test and have a whole litany of tests fire off. Right now, it means we can make sure that our front end does the right thing of accepting correct code and rejecting incorrect code. In the future it means we can test the whole compiler.

Let's also keep our tests together in one place, separate from the compiler source code. I created a new directory named tests/ to hold all the test files. Inside the new directory, I will create a shell script that will handle running all the tests named test.sh. This shell script will be quite flexible and allow us to create an arbitrary number of tests, all of which will be automatically incorporated into the testing framework once created. This shell script looks like this:

#!/bin/sh

echo PL/0 compiler test suite
echo ========================

for i in *.pl0 ; do
  /usr/bin/printf "%.4s... " $i
  ../pl0c $i > /dev/null 2>&1
  if [ $? -eq 0 ] ; then
    echo ok
  else
    echo fail
  fi
done

That's all we need for now. This allows us to have up to 10,000 tests, named and numbered 0000 to 9999. If for any reason we need more than that, we can alter the printf line to read %.5s and give ourselves up to 100,000 tests. Of course, you don't need to follow my numbering scheme, but it's how my brain organizes things best.

Writing our first tests

I am going to write a bunch of basic tests to start us off. The first test is the empty program, the smallest legal PL/0 program. That would be a single dot. So the 0000.pl0 program will contain . and nothing more.

For all other tests, I like to begin with a comment containing the test number and a small description of what the test is. For 0001.pl0, I want to make sure that all types of comments work. The program itself will still be the empty program but it will be decorated with comments. It looks like this:

{ 0001: Comments }
{
  Multi-line comment
}
. { Inline comment }
{ End of program comment }

Now, running make test will automatically run both tests without having to teach the shell script about the tests themselves. Neat! That makes our lives easier. I added a total of ten tests to get started. All the tests I wrote are expected-pass tests. There are ways to have expected-fail tests as well, such as having tests 0000-0999 be expected-pass and tests 1000-1999 be expected-fail.

Adding your own tests

Feel free to submit your own tests to the GitHub repository. Please make sure that if you submit tests:

1. If you didn't write the test, it has an open source license
2. If you did write the test, you agree to release it under an open source license

On the next episode: A code generator

Now we can turn our attention to the code generator. Our initial bootstrap compiler will be completed once the code generator is finished! Well, almost. We will write a code generator for the PL/0 language as we have it. But you may have already noticed that there are some missing bits to be fully useful, like basic input and output. We'll write a code generator for the language we have first, then we'll consider adding extras.

OpenBSD httpd