Using makefiles for regression testing
by Mithrandir
Sometimes, I found myself starting a large coding project (more than 100-150 lines) with many complex tests. In this case, I start the development phase by first creating a makefile to build all the sources along with creating a svn repository for them, even if this repository is to be kept locally. Exactly like everyone else.
But wait, my makefiles contain two more targets. One of them is simply used for documenting the code. To put it simply, I have become addicted to seeing a doxygen documentation of my own code when it grows upper than a certain limit. Instead of switching through the source files and scrolling through them, I prefer to see the globals and the definitions in firefox. This is understandable. Of course, the doxygen autoconfig file is added to the repo along with the other sources.
The second target is what makes my makefile somehow special. It is a target used for testing purposes: valgrind, vimdiff and many tests files are used. After completing three college assignment successfully by using this method, I can guarantee that all projects will start in the same way.
Enough with the chat, let’s see one example:
.PHONY: all clean test doc TARGET = ./MMRP OBJS = MMRP.o globals.o purificare.o invarianti.o CC = gcc CFLAGS = -Wall -O3 #-g -O0 DEBUG = #valgrind FILEI = in01.txt FILEP = in01p.txt FILEPC = inp01.txt FILEO = out1.txt FILEOC = out01.txt FILEARGS = $(FILEI) $(FILEP) $(FILEO) FILEDEL = $(FILEP) $(FILEO) TESTDIR = Teste/ all: $(TARGET) $(TARGET): $(OBJS) clean: rm -f $(OBJS) $(TARGET) $(FILEDEL) rm -rf html cd $(TESTDIR) && make clean EXECFILE=$(TARGET) test: $(TARGET) $(DEBUG) $(TARGET) $(FILEARGS) echo "Verificare retea purificata" vimdiff $(FILEP) $(FILEPC) || echo "Verificare invarianti" vimdiff $(FILEO) $(FILEOC) || echo "Test done" echo "" echo "Testing public tests" cp $(TARGET) $(TESTDIR) cd $(TESTDIR) && make DEBUG=$(DEBUG) EXECFILE=$(TARGET) doc: doxygen
where the second makefile, the one inside the tests page is
.PHONY: all clean TARGET = test EXECFILE = #./MMRP DEBUG = #valgrind TESTINPS = tests TESTOUPS = outs INPF = inp.txt OUTF = outp.txt NUMTESTS = 8 all: $(TARGET) $(TARGET): for i in `seq -f "%02g" 1 $(NUMTESTS)`; do \ echo "Test $$i";\ $(DEBUG) $(EXECFILE) $(TESTINPS)/in$$i.txt $(INPF) $(OUTF); \ vimdiff $(INPF) $(TESTOUPS)/inp$$i.txt;\ vimdiff $(OUTF) $(TESTOUPS)/out$$i.txt;\ done clean: rm -f $(EXECFILE) $(INPF) $(OUTF)
Having learnt the trick os second order argument expansion (the $$i in the second makefile), testing my programs is as simple as writting new tests for them. By using Valgrind and by enabling the -g compiling switch, I am sure not to allow any bugs propagate in my programs.
Of course, unit testing can be done via different libraries but, for C, I prefer this method. In Python I ussualy use the unittest module, it is far easier.
Another use for this makefile trick (the one with the tests) is in automatic checking of assignments. I’ve used it when I had to check my students assignments or when I reviewed the submissions for my programming contest. To put it simply, I had a script which will simply hijack the makefile, adding the needed target and then running make test. Simple, isn’t it?
