(version originale en français ici)
In the two previous posts, we started the development of a dynamic library on Linux: the first one saw us building the library and managing version numbers using symbolic links, in the second one we traced library calls and did step-by-step debugging. Now we are interested in checking the coverage of the library.
Code coverage is a measure indicating the percentage of lines of code that were actually covered during a program execution. We can gradually expand the set of tests to obtain 100% coverage (and ensure that the code has been fully verified).
The best known tool under Linux for code coverage is gcov, which instruments the source code and provide us detailed statistics after execution. It is easy to use it to verify coverage of a source file compiled into an executable. We will use it for library code which requires some more attention.
Compilation
Let’s start with the same files and directories (grouped in this archive) as in the previous articles. A directory named « factorial » contains four subdirectories: src, include and lib respectively where the source files, header files and compiled files of the library are. The fourth sub-directory « test » contains the source and executable files of a program using our library libfact.
[~]$ cd factorial/ [factorial]$ ls include lib src test [factorial]$ ls include/ fact.h [factorial]$ ls lib/ libfact.so libfact.so.2 libfact.so.2.0 [factorial]$ ls src/ fact.c [factorial]$ ls test/ factorial.c [factorial]$
NB: The library was already compiled in the above example, but we will rebuild it.
At first we will compile the library code, as in the first article, but adding the --coverage option of gcc. This option has two different roles:
- During compilation, it has the same meaning as
-fprofile-arcsand-ftest-coverageoptions (which were used with the previous versions ofgcc): the first one added instrumentation data to the executable code (counters), the second one created a table of correspondence between instructions blocks and lines of source code (in a file named after the source file with the extension.gcno) - During linking, it is equivalent to
-lgcov(which was added automatically by-ftest-coverage) that incorporates the necessary entry points for the subsequent use ofgcov.
Here’s the compilation of our library.
[factorial]$ gcc -c --coverage -fPIC -I include/ -o ./src/fact.o ./src/fact.c [factorial]$ ls src/ fact.c fact.gcno fact.o [factorial]$
We see that with the --coverage option, the compilation generated, in addition to the fact.o object file, a fact.gcno file, containing the relationships between the blocks of code and the line numbers. We continue.
[factorial]$ gcc -shared -I include/ -Wl,-soname,libfact.so.2 -o lib/libfact.so.2.0 ./src/fact.o --coverage [factorial]$ ls -l lib/ total 20 lrwxrwxrwx 1 cpb cpb 12 2012-02-11 13:39 libfact.so -> libfact.so.2 lrwxrwxrwx 1 cpb cpb 14 2012-02-11 13:39 libfact.so.2 -> libfact.so.2.0 -rwxrwxr-x 1 cpb cpb 16866 2012-02-11 17:10 libfact.so.2.0 [factorial]$
We have rebuilt the libfact.so.2.0 library. Symbolic links are used to manage the major and minor version numbers, as we saw in the first article. Now compile an executable file, without --coverage option (or use the executable file of previous articles).
[factorial]$ gcc -I include/ -L lib/ -o test/factorial test/factorial.c -lfact [factorial]$ ls test/ factorial factorial.c [factorial]$
Execution
Program execution takes place quite normally (although in practice it is slightly slower). We must, of course, set the environment variable LD_LIBRARY_PATH to specify where the dynamic linker will find the library needed to run the application.
[factorial]$ export LD_LIBRARY_PATH=lib/ [factorial]$ test/factorial 4 5 6 4! = 24 5! = 120 6! = 720 [factorial]$ ls src/ fact.c fact.gcda fact.gcno fact.o [factorial]$
A new file named fact.gcda appeared, containing the execution statistics for fact.gcno blocks of code (and block transitions).
Results
For information on the code coverage of a source file, we invoke gcov indicating the source file name. The results are computed independently for each source file of the application (or library).
We will use the -o option of gcov to specify the directory name for the .c, .gcno and .gcda files.
[factorial]$ gcov -o src/ fact.c File 'src/fact.c' Lines executed:87.50% of 8 src/fact.c:creating 'fact.c.gcov' [factorielle]$
gcov tells us that we have only performed 87.5% of the eight lines of code in our function. What happened? We see that gcov also created a file named « fact.c.gcov » in which he puts a copy of our source code, numbering the lines, adding a header and a column of statistics at the beggining of the line.
[factorial]$ ls
fact.c.gcov include lib src test
[factorial]$ cat fact.c.gcov
-: 0:Source:src/fact.c
-: 0:Graph:src/fact.gcno
-: 0:Data:src/fact.gcda
-: 0:Runs:1
-: 0:Programs:1
-: 1:#include
-: 2:
3: 3:int factorial(long int n, long long int * result)
-: 4:{
3: 5: * result = 1;
3: 6: if (n < 0)
#####: 7: return -1;
-: 8: do {
12: 9: (*result) = (*result) * n;
12: 10: n = n - 1;
12: 11: } while (n > 1);
3: 12: return 0;
-: 13:}
-: 14:
[factorial]$
The header describes the files involved and the number of executions (only one here). The left column shows the number of passes on each line. Lines containing a dash « – » do not match any compiled code. We see that lines 3, 5, 6 and 12 were scanned three times (one invocation for each argument on the command line), and that lines 9, 10 and 11 were executed 12 times (iterations to calculate factorial).
If we repeat the operation, the counters are cumulated.
[factorial]$ test/factorial 3 8
3! = 6
8! = 40320
[factorial]$ gcov -o src/ fact.c
File 'src/fact.c'
Lines executed:87.50% of 8
src/fact.c:creating 'fact.c.gcov'
[factorial]$ cat fact.c.gcov
-: 0:Source:src/fact.c
-: 0:Graph:src/fact.gcno
-: 0:Data:src/fact.gcda
-: 0:Runs:2
-: 0:Programs:1
-: 1:#include
-: 2:
5: 3:int factorial(long int n, long long int * result)
-: 4:{
5: 5: * result = 1;
5: 6: if (n < 0)
#####: 7: return -1;
-: 8: do {
21: 9: (*result) = (*result) * n;
21: 10: n = n - 1;
21: 11: } while (n > 1);
5: 12: return 0;
-: 13:}
-: 14:
[factorial]$
But, what about line 7? Why these « #####« ?
Unlike spreadsheet programs, this symbol does not mean that the number is too large to fit in the column, but that the line (which corresponds to compiled code) was never executed. Two advantages with this notation:
- it attracts the eye better than a single « 0 » would do,
- it allows us to do an automated search of unexecuted lines using
grep.
Let’s see:
[factorial]$ gcov -o src fact.c
File 'src/fact.c'
Lines executed:87.50% of 8
src/fact.c:creating 'fact.c.gcov'
[factorial]$ grep '#####' fact.c.gcov
#####: 7: return -1;
[factorial]$
The line number (7) being printed with the content, we can have a quick overview of non executed code.
Correction
We have detected a problem with our test library, since a branch was never executed. This may be due to different reasons.
- An incomplete tests set. This is the case here and we will fix it easily below.
- Legacy code that is never invoked again (dead code). It is important to make it disappear because it disrupts the maintenance of the program.
- Lines of code used to handle rare error cases, difficult to test. We will deal with this problem in the next paragraph.
Here the solution is simple: the uncovered line corresponds to an invocation of the function with a negative argument. It is easy to test.
[factorial]$ ./test/factorial -1 -1! doesn't exist [factorial]$ gcov -o src fact.c File 'src/fact.c' Lines executed:100.00% of 8 src/fact.c:creating 'fact.c.gcov' [factorial]$
Now gcov tells us that all the lines of our program have been covered by our test set. This reduces the probability of remaining bug.
Error handling
A major difficulty to achieve 100% code coverage during software testing is to validate the behavior in case of system error.
Take a look at a well-known system call: malloc(). We asked him to allocate a memory area of a certain size (given in bytes) and he returns a pointer. All documentation tell you that in case of lack of memory, malloc() returns a NULL pointer. (Although this case is particularly difficult to produce with Linux, we will discuss this in a future article). Also, the conscientious programmer will write something like.
char * buffer;
buffer = malloc(BUFFER_SIZE);
if (buffer == NULL) {
display_error("Insufficient memory");
register_error_code(-ENOMEM);
return -1;
}
// ...
Unfortunately the lines in between braces are difficult to test because we can not « force » malloc() to fail. The failure circumstances are based on too many parameters external to the application to be reproducible.
In the specific case of malloc(), the Glibc library provides us entry points that can be used to replace the function – see malloc_hook(3) – but it is not possible for other system calls.
However there are several solutions. One of them, I have used several times, is to use a software layer that replicates the minimum system calls we need and simulates a failure if certain criteria are met. For example the following routine reproduces malloc() but fails after a number of invocations contained in the environment variable MALLOC_FAIL.
src/my_malloc.c:
#include <stdio.h>
#include <stdlib.h>
void * my_malloc(size_t length)
{
char * string;
char buffer[32];
int count;
string = getenv ("MALLOC_FAIL");
if (string != NULL) {
if (sscanf(string, "%d", & count) == 1) {
count --;
if (count == 0)
return NULL;
snprintf(buffer, 80, "%d", count);
setenv("MALLOC_FAIL", buffer, 1 );
}
}
return malloc(length);
}
We can declare it in a header file as follows:
include/my_malloc.h:
#ifndef MY_MALLOC_H
#define MY_MALLOC_H
#ifndef NDEBUG
extern void * my_malloc(size_t);
#else
#define my_malloc(L) malloc(L)
#endif
#endif
In this way, depending on the presence or absence of the NDEBUG constant, which traditionally represents for the C library the production version of the code, our routine will be compiled as usual malloc() or with our management of the environment variable.
Compile a dynamic library with our minimal abstraction layer minimum.
[factorial]$ gcc -c -fPIC -Wall -I include/ -o src/my_malloc.o src/my_malloc.c [factorial]$ gcc -shared -Wl,-soname,libmytest.so.1 -o lib/libmytest.so.1.0 src/my_malloc.o [factorial]$ ldconfig -n lib/ [factorial]$ ln -sf libmytest.so.1 lib/libmytest.so [factorial]$ ls -l lib/libmy* lrwxrwxrwx 1 cpb cpb 14 2012-02-12 04:12 lib/libmytest.so -> libmytest.so.1 lrwxrwxrwx 1 cpb cpb 16 2012-02-12 04:11 lib/libmytest.so.1 -> libmytest.so.1.0 -rwxrwxr-x 1 cpb cpb 7014 2012-02-12 04:11 lib/libmytest.so.1.0 [factorial]$
Let’s write a small program that uses our test library by looping around my_malloc().
test/test-mymalloc.c:
#include <stdio.h>
#include <stdlib.h>
#include <my_malloc.h>
int main(void)
{
int i = 1;
while(1) {
fprintf(stderr, "i = %2d...", i);
if (my_malloc(10) == NULL) {
fprintf(stderr, "failure!n");
break;
}
fprintf(stderr, "okn");
i ++;
}
return 0;
}
Compilation…
[factorial]$ gcc -I include/ -L lib/ -Wall -o test/test-mymalloc test/test-mymalloc.c -lmytest [factorial]$
First test, without the environment variable.
[factorial]$ unset MALLOC_FAIL [factorial]$ ./test/test-mymalloc i = 1...ok i = 2...ok i = 3...ok i = 4...ok i = 5...ok [...] i = 21928...ok i = 21929...ok i = 21930...ok i = 21931...ok i = 21932... (Control-C) [factorial]$
Of course, our program does not stop. Or rather it will stop after a long runtime when exhausting its 3GB address space (on a 32-bits machine).
Try again by forcing a failure in the fourth malloc() call.
[factorial]$ export MALLOC_FAIL=4 [factorial]$ ./test/test-mymalloc i = 1...ok i = 2...ok i = 3...ok i = 4...failure! [factorial]$
Of course, this principle of forcing system call failures under the control of an environment variable – or other parameters (global variable, file, shared memory area, etc.) – can be applied equally to a library code when you need 100% code coverage over the entire set of tests for an application.
Conclusion
We observed in this small series of articles, how to create, debug and test a dynamic library. I encourage you to do your own tests, referring to the documentation of gcc, gdb, gcov, but also other complementary tools such as gprof, ldconfig, valgrind, etc.
All comments, remarks, corrections, etc.. are welcome.
