The ltrace and strace commands are useful, but it would be really nice to trace both user-space and system-space function calls. The -S option to ltrace will do this. The next example shows the ltrace -S output from the previous executable:

$> ltrace -S ./a.out
SYS_brk(NULL)                                            = 0xa9f000
SYS_mmap(0, 4096, 3, 34, 0xffffffff)                     = 0x7fcdce4ce000
SYS_access(0x3b0dc1d380, 4, 0x3b0dc00158, 0, 0)          = -2
SYS_open("/etc/ld.so.cache", 0, 01)                      = 4
SYS_fstat(4, 0x7ffd70342bc0, 0x7ffd70342bc0, 0, 0xfefefefefefefeff) = 0
SYS_mmap(0, 95195, 1, 2, 4)                              = 0x7fcdce4b6000
SYS_close(4)                                             = 0
SYS_open("/lib64/libc.so.6", 0, 00)                      = 4
SYS_read(4, "\177ELF\002\001\001\003", 832)              = 832
SYS_fstat(4, 0x7ffd70342c20, 0x7ffd70342c20, 4, 0x7fcdce4ce640) = 0
SYS_mmap(0x3b0e000000, 0x393928, 5, 2050, 4)             = 0x3b0e000000
SYS_mprotect(0x3b0e18a000, 0x200000, 0, 1, 4)            = 0
SYS_mmap(0x3b0e38a000, 24576, 3, 2066, 4)                = 0x3b0e38a000
SYS_mmap(0x3b0e390000, 14632, 3, 50, 0xffffffff)         = 0x3b0e390000
SYS_close(4)                                             = 0
SYS_mmap(0, 4096, 3, 34, 0xffffffff)                     = 0x7fcdce4b5000
SYS_mmap(0, 4096, 3, 34, 0xffffffff)                     = 0x7fcdce4b4000
SYS_mmap(0, 4096, 3, 34, 0xffffffff)                     = 0x7fcdce4b3000
SYS_arch_prctl(4098, 0x7fcdce4b4700, 0x7fcdce4b3000, 34, 0xffffffff) = 0
SYS_mprotect(0x3b0e38a000, 16384, 1, 0x3b0de20fd8, 0x1f25bc2) = 0
SYS_mprotect(0x3b0de1f000, 4096, 1, 0x4003e0, 0x1f25bc2) = 0
(0, 0, 987392, -1, 0x1f25bc2)                            = 0x3b0de21160
SYS_munmap(0x7fcdce4b6000, 95195)                        = 0
__libc_start_main(0x4005fe, 1, 0x7ffd703435c8, 0x400660, 0x400650 <unfinished ...>
malloc(100 <unfinished ...>
SYS_brk(NULL)                                            = 0xa9f000
SYS_brk(0xac0000)                                        = 0xac0000
<... malloc resumed> )                                   = 0xa9f010
strcat("", "testing")                                    = "testing"
puts("testing" <unfinished ...>
SYS_fstat(1, 0x7ffd70343370, 0x7ffd70343370, 0x7ffd70343230, 0x3b0e38f040) = 0
SYS_mmap(0, 4096, 3, 34, 0xffffffff)                     = 0x7fcdce4cd000
SYS_write(1, "testing\n", 8)                             = 8
<... puts resumed> )                                     = 8
free(0xa9f010)                                           = <void>
exit(0 <unfinished ...>
SYS_exit_group(0 <no return ...>
+++ exited (status 0) +++

This shows the same type of startup call (sbrk, mmap, and so on) as the strace example.

When a user-space function invokes a system-space function (as with the malloc and puts calls), the display shows that the user-space function was interrupted (malloc(100 <unfinished...>) and then resumed (<... malloc resumed>) after the system call was completed.

Note that the malloc call needed to pass control to sbrk to allocate more memory for the application. However, the free call does not shrink the application; it just frees the memory for future use by this application.