(Version originale en français)
In the previous article we saw how to build a dynamic library and properly manage its major and minor version numbers for easy maintenance, both for the developer (of the library but also of the applications that use it) and for the administrator of the system where it is installed. We will now consider how to debug our library and applications that call it.
Calls tracking
The first tool to know is ltrace. It sends to the error output of the process a log of all function calls to dynamic libraries. We put ourselves back in the same situation as for the previous article with the following directories:
factorial/srccontains the source code of the libraryfactorial/includewhere the header files of the library arefactorial/libcontaining the compiled files of the libraryfactorial/testwhere we find source codes and executable files of the applications using the library
Here is an overview of the content of our directory:
[~] cd factorial [factorial]$ ls include lib src test [factorial]$ ls include/ fact.h [factorial]$ ls -l lib/ total 8 lrwxrwxrwx 1 cpb cpb 12 2012-02-04 05:04 libfact.so -> libfact.so.2 lrwxrwxrwx 1 cpb cpb 14 2012-02-04 05:04 libfact.so.2 -> libfact.so.2.0 -rwxrwxr-x 1 cpb cpb 6661 2012-02-04 05:04 libfact.so.2.0 [factorial]$ ls src/ fact.c fact.o [factorial]$ ls test/ factorial factorial.c [factorial]$
Details to create executable files and symbolic links were in the previous article. Here is an example of the test program.
[factorial]$ export LD_LIBRARY_PATH=lib/ [factorial]$ ./test/factorial 7 7! = 5040 [factorial]$
Let’s use ltrace to see library function calls.
[factorial]$ ltrace ./test/factorial 4 5 6 __libc_start_main(0x80485b4, 4, 0xbf896624, 0x80486b0, 0x8048720 __isoc99_sscanf(0xbf89745f, 0x8048796, 0xbf896578, 0x80486d1, 0x8048500) = 1 factorial(4, 0xbf896570, 0xbf896578, 0x80486d1, 0x8048500) = 0 fprintf(0xb7896500, "%ld! = %lldn", 4, ...4! = 24 ) = 8 __isoc99_sscanf(0xbf897461, 0x8048796, 0xbf896578, 24, 0) = 1 factorial(5, 0xbf896570, 0xbf896578, 24, 0) = 0 fprintf(0xb7896500, "%ld! = %lldn", 5, ...5! = 120 ) = 9 __isoc99_sscanf(0xbf897463, 0x8048796, 0xbf896578, 120, 0) = 1 factorial(6, 0xbf896570, 0xbf896578, 120, 0) = 0 fprintf(0xb7896500, "%ld! = %lldn", 6, ...6! = 720 ) = 9 +++ exited (status 0) +++ [factorial]$
First of all there is a mix between the standard output display and the error output content, making the reports difficult to read. We will send error output to a file:
[factorial]$ ltrace ./test/factorial 4 5 6 2>traces.txt 4! = 24 5! = 120 6! = 720 [factorial]$
If our process uses its error output, we can ask ltrace to put its messages directly into a file, using the -o option.
[factorial]$ ltrace -o traces.txt ./test/factorial 4 5 6 4! = 24 5! = 120 6! = 720 [factorial]$
Let’s see the content of our trace file.
[factorial]$ cat traces.txt __libc_start_main(0x80485b4, 4, 0xbf9384f4, 0x80486b0, 0x8048720 <unfinished ...> _isoc99_sscanf(0xbf93a45f, 0x8048796, 0xbf938448, 0x80486d1, 0x8048500) = 1 factorial(4, 0xbf938440, 0xbf938448, 0x80486d1, 0x8048500) = 0 fprintf(0xb771d500, "%ld! = %lldn", 4, ...) = 8 __isoc99_sscanf(0xbf93a461, 0x8048796, 0xbf938448, 24, 0) = 1 factorial(5, 0xbf938440, 0xbf938448, 24, 0) = 0 fprintf(0xb771d500, "%ld! = %lldn", 5, ...) = 9 __isoc99_sscanf(0xbf93a463, 0x8048796, 0xbf938448, 120, 0) = 1 factorial(6, 0xbf938440, 0xbf938448, 120, 0) = 0 fprintf(0xb771d500, "%ld! = %lldn", 6, ...) = 9 +++ exited (status 0) +++ [factorial]$
Indeed we see the call to our factorial() function, but there is something strange: ltrace prints five parameters for our function, but there are in fact only two parameters (see fact.h).
In fact, ltrace relies on a configuration file named /etc/ltrace.conf that contains the number and type of arguments of the functions of the system dynamic libraries. If it does not find the function in this file, it displays five arguments by default.
/etc/ltrace.conf ; ltrace.conf ; ; ~/.ltrace.conf will also be read, if it exists. The -F option may be ; used to suppress the automatic inclusion of both this file and ; ~/.ltrace.conf, and load a different config file or config files ; instead. [...] ; arpa/inet.h int inet_aton(string,addr); string inet_ntoa(addr); ; It isn't an ADDR but an hexa number... addr inet_addr(string); [...] ; stdio.h int fclose(file); int feof(file); int ferror(file); int fflush(file); char fgetc(file); addr fgets(+string, int, file); int fileno(file); file fopen(string,string); file fopen64(string,string); int fprintf(file,format); int fputc(char,file); int fputs(string,file); ulong fread(addr,ulong,ulong,file); ulong fread_unlocked(addr,ulong,ulong,file); ulong fwrite(string,ulong,ulong,file); ulong fwrite_unlocked(string,ulong,ulong,file); int pclose(addr); void perror(string); addr popen(string, string); int printf(format); int puts(string); int remove(string); int snprintf(+string2,ulong,format); int sprintf(+string,format); [...] int SYS_waitpid(int,addr,int); ulong SYS_readv(int,addr,int); ulong SYS_writev(int,addr,int); int SYS_mprotect(addr,int,int); int SYS_access(string,octal);
Of course our function does not appear in this file. We could change it (if the library was installed in a location accessible to all users), but I propose instead to create an additional file .ltrace.conf that we put in our home directory (where ltrace look for additional configuration files).
[factorial]$ cat ~/.ltrace.conf int factorial(long,addr); [factorial]$ ltrace -o traces.txt ./test/factorial 4 5 6 4! = 24 5! = 120 6! = 720 [factorial]$ cat traces.txt __libc_start_main(0x80485b4, 4, 0xbf8b3764, 0x80486b0, 0x8048720 __isoc99_sscanf(0xbf8b545c, 0x8048796, 0xbf8b36b8, 0x80486d1, 0x8048500) = 1 factorial(4, 0xbf8b36b0) = 0 fprintf(0xb7704500, "%ld! = %lldn", 4, ...) = 8 __isoc99_sscanf(0xbf8b545e, 0x8048796, 0xbf8b36b8, 24, 0) = 1 factorial(5, 0xbf8b36b0) = 0 fprintf(0xb7704500, "%ld! = %lldn", 5, ...) = 9 __isoc99_sscanf(0xbf8b5460, 0x8048796, 0xbf8b36b8, 120, 0) = 1 factorial(6, 0xbf8b36b0) = 0 fprintf(0xb7704500, "%ld! = %lldn", 6, ...) = 9 +++ exited (status 0) +++ [factorial]$
We may also use the -F option to specify which ltrace.conf file to use.
This time the result is perfect, we see our function calls, with the value to compute, the address of the result to complete and the return status (0 means « all right »).
Using ltrace, and its cousin command strace for system calls, is very usefull when developping applications and library. Non-intrusive (no special compiler option) and no requirement for source code make it usable even on programs deliveres in binary form only. I remember having used it successfully a few years ago on an application that needed a configuration file, but the documentation referred to an invalid directory. So I started ltrace on the application without providing the configuration file. Of course the application failed to start, but I could find in the logs (filtered with grep) the calls to the fopen() library function, and see the sucessive attempts before failure (something like ~/.APPLICATION/, /usr/lib/APPLICATION, /etc/APPLICATION …)
Debugging with Gdb
In most cases during application development, we consider library calls as invocations of elementary functions on whose contents we do not question. However, dynamic libraries may also require development and step-by-step analysis.
So, let’s try using gdb with the executable file we produced in the previons article.
[factorial]$ gdb ./test/factorial GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /home/cpb/factorial/test/factorial...(no debugging symbols found)...done. (gdb)
The debugger has found our executable and loaded it into memory. However, he can’t find any symbol table needed for debugging. Let’s just try to put a breakpoint at the start of main function.
(gdb) break main Breakpoint 1 at 0x80485b9 (gdb)
It works. Run the program with a value argument.
(gdb) run 5 Starting program: /home/cpb/factorial/test/factorial 5 Breakpoint 1, 0x080485b9 in main () (gdb)
We stopped in the beginning of main(). Try to move one step forward.
(gdb) next Single stepping until exit from function main, which has no line number information. 5! = 120 0xb7e62113 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6 (gdb)
Indeed, the debugger has no notion of line of code, it goes in one run until the end of main(). We can only allow the process to complete and exit the debugger.
(gdb) cont Continuing. [Inferior 1 (process 15847) exited normally] Undefined command: "exit". Try "help". (gdb) quit [factorial]$
Now, we compile our code with the -g option, in order to insert in the executable file a table of associations between memory addresses and lines of code. Then repeat the experience.
[factorial]$ gcc -I include/ -L lib/ -o test/factorial test/factorial.c -l fact -g
[factorial]$ gdb ./test/factorial
GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>
Reading symbols from /home/cpb/factorial/test/factorial...done.
(gdb) break main
Breakpoint 1 at 0x80485bf: file test/factorial.c, line 11.
(gdb) run 5
Starting program: /home/cpb/factorial/test/factorial 5
Breakpoint 1, main (argc=2, argv=0xbffff274) at test/factorial.c:11
11 if (argc < 2) {
(gdb)
This time, we note that gdb displays correctly the line 11. Continue to move forward instruction-by-instruction with the step command.
(gdb) step
15 for (i = 1; i < argc; i ++)
(gdb) step
16 if (sscanf(argv[i], "%ld", & n) == 1) {
(gdb) step
17 if (factorial(n, & f) == 0)
(gdb) step
18 fprintf(stdout, "%ld! = %lldn", n, f);
(gdb)
Unfortunately gdb does not have the sources of the library, so we can not enter the factorial() function, no more than we could in sscanf() or fprintf().
(gdb) step 5! = 120 15 for (i = 1; i < argc; i ++) (gdb) step 22 return EXIT_SUCCESS; (gdb) step 23 } (gdb) step 0xb7e62113 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6 (gdb) step Single stepping until exit from function __libc_start_main, which has no line number information. [Inferior 1 (process 27455) exited normally] (gdb) quit [factorial]$
Debugging the library
We have to compile the library with the -g option.
[factorial]$ gcc -I include/ -o src/fact.o -c src/fact.c -g [factorial]$ gcc -shared -I include/ -Wl,-soname,libfact.so.2 -o lib/libfact.so.2.0 src/fact.o [factorial]$ gcc -I include/ -L lib/ -o test/factorial test/factorial.c -l fact -g
And start debugging…
[factorial]$ gdb ./test/factorial
GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/cpb/factorial/test/factorial...done.
(gdb) break main
Breakpoint 1 at 0x80485bf: file test/factorial.c, line 11.
(gdb) run 5
Starting program: /home/cpb/factorial/test/factorial 5
(gdb) break main
Breakpoint 1, main (argc=2, argv=0xbffff274) at test/factorial.c:11
11 if (argc < 2) {
(gdb) step
15 for (i = 1; i < argc; i ++)
(gdb) step
16 if (sscanf(argv[i], "%ld", & n) == 1) {
(gdb) step
17 if (factorial(n, & f) == 0)
(gdb) step
factorial (n=5, result=0xbffff1c0) at src/fact.c:5
5 * result = 1;
(gdb)
Now we really are into our factorial() function, let’s continue step-by-step a few instructions more…
(gdb) step 6 if (n < 0) (gdb) step 9 (*result) = (*result) * n; (gdb) step 10 n = n - 1; (gdb) step 11 } while (n > 1); (gdb) step 9 (*result) = (*result) * n; (gdb) step 10 n = n - 1; (gdb)
We can also look at the state of the variables. The gdb printing of the results is a bit surprising at first because it prefixes the evaluated terms by a $ character followed by a serial number. This allows to easily write graphical front-ends (like ddd, xxgdb, Eclipse, etc..) that retrieve the returned values.
(gdb) print *result $1 = 20 (gdb) print n $2 = 4 (gdb) cont Continuing. 5! = 120 [Inferior 1 (process 30206) exited normally] (gdb) quit [factorial]$
Source files locatoin
The source files location is stored into the executable file when compiling with the -g option. However they can be moved or library can be compiled on another machine that those used for debugging. We must therefore have a way to tell gdb where to find the source files of the library. Do a test by moving the sources from the library into a totally different directory.
[factorial]$ mkdir -p ~/tmp/sources [factorial]$ mv src/* ~/tmp/sources/ [factorial]$ gdb ./test/factorial GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /home/cpb/factorial/test/factorial...done. (gdb) break factorial Breakpoint 1 at 0x80484f0 (gdb) run 5 Starting program: /home/cpb/factorial/test/factorial 5 Breakpoint 1, factorial (n=5, result=0xbffff1c0) at src/fact.c:5 5 src/fact.c: No such file or directory. in src/fact.c (gdb) quit A debugging session is active. Inferior 1 [process 31536] will be killed. Quit anyway? (y or n) y [factorial]$
Of course the debugging failed. We will now try the directory command at the gdb prompt.
[factorial]$ gdb ./test/factorial GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /home/cpb/factorial/test/factorial...done. (gdb) directory ~/tmp/sources/ Source directories searched: /home/cpb/tmp/sources:$cdir:$cwd (gdb) break factorial Breakpoint 1 at 0x80484f0 (gdb) run 5 Starting program: /home/cpb/factorial/test/factorial 5 Breakpoint 1, factorial (n=5, result=0xbffff1c0) at src/fact.c:5 5 * result = 1; (gdb) step 6 if (n < 0) (gdb) step 9 (*result) = (*result) * n; (gdb) cont Continuing. 5! = 120 [Inferior 1 (process 31845) exited normally] (gdb) quit [factorial]$
It works perfectly.
Conclusion
We have seen two important steps in the development of a library: function calls tracking and step-by-step debugging. We will examine the coverage tests in the next article.
