Using makefiles for regression testing

by Mithrandir

Sometimes, I found myself starting a large coding project (more than 100-150 lines) with many complex tests. In this case, I start the development phase by first creating a makefile to build all the sources along with creating a svn repository for them, even if this repository is to be kept locally. Exactly like everyone else.

But wait, my makefiles contain two more targets. One of them is simply used for documenting the code. To put it simply, I have become addicted to seeing a doxygen documentation of my own code when it grows upper than a certain limit. Instead of switching through the source files and scrolling through them, I prefer to see the globals and the definitions in firefox. This is understandable. Of course, the doxygen autoconfig file is added to the repo along with the other sources.

The second target is what makes my makefile somehow special. It is a target used for testing purposes: valgrind, vimdiff and many tests files are used. After completing three college assignment successfully by using this method, I can guarantee that all projects will start in the same way.

Enough with the chat, let’s see one example:

.PHONY: all clean test doc

TARGET = ./MMRP
OBJS = MMRP.o globals.o purificare.o invarianti.o
CC = gcc
CFLAGS = -Wall -O3 #-g -O0
DEBUG = #valgrind
FILEI = in01.txt
FILEP = in01p.txt
FILEPC = inp01.txt
FILEO = out1.txt
FILEOC = out01.txt
FILEARGS = $(FILEI) $(FILEP) $(FILEO)
FILEDEL = $(FILEP) $(FILEO)
TESTDIR = Teste/

all: $(TARGET)

$(TARGET): $(OBJS)

clean:
    rm -f $(OBJS) $(TARGET) $(FILEDEL)
    rm -rf html
    cd $(TESTDIR) && make clean EXECFILE=$(TARGET)

test: $(TARGET)
    $(DEBUG) $(TARGET) $(FILEARGS)
    echo "Verificare retea purificata"
    vimdiff $(FILEP) $(FILEPC) || echo "Verificare invarianti"
    vimdiff $(FILEO) $(FILEOC) || echo "Test done"
    echo ""
    echo "Testing public tests"
    cp $(TARGET) $(TESTDIR)
    cd $(TESTDIR) && make DEBUG=$(DEBUG) EXECFILE=$(TARGET)

doc:
    doxygen

where the second makefile, the one inside the tests page is

.PHONY: all clean

TARGET = test
EXECFILE = #./MMRP
DEBUG = #valgrind
TESTINPS = tests
TESTOUPS = outs
INPF = inp.txt
OUTF = outp.txt
NUMTESTS = 8

all: $(TARGET)

$(TARGET):
    for i in `seq -f "%02g" 1 $(NUMTESTS)`; do \
        echo "Test $$i";\
        $(DEBUG) $(EXECFILE) $(TESTINPS)/in$$i.txt $(INPF) $(OUTF); \
        vimdiff $(INPF) $(TESTOUPS)/inp$$i.txt;\
        vimdiff $(OUTF) $(TESTOUPS)/out$$i.txt;\
    done

clean:
    rm -f $(EXECFILE) $(INPF) $(OUTF)

Having learnt the trick os second order argument expansion (the $$i in the second makefile), testing my programs is as simple as writting new tests for them. By using Valgrind and by enabling the -g compiling switch, I am sure not to allow any bugs propagate in my programs.

Of course, unit testing can be done via different libraries but, for C, I prefer this method. In Python I ussualy use the unittest module, it is far easier.

Another use for this makefile trick (the one with the tests) is in automatic checking of assignments. I’ve used it when I had to check my students assignments or when I reviewed the submissions for my programming contest. To put it simply, I had a script which will simply hijack the makefile, adding the needed target and then running make test. Simple, isn’t it?

About these ads