Library Call Interception

In the context of system analysis and system trouble-shooting the tracing and interception of individual function calls, e.g. system calls, from user-space processes might be required or at least useful. When your system is running an up-to-date version of Linux, probing could be applied by using SystemTap or on a more specialized scale malloc hooks for functions of the malloc family. This post shows a Unix-generic solution to this problem relying on symbol overloading and pre-loading of shared libraries at runtime. While this approach is not tailored to Linux, the examples however are compiled and executed on an Ubuntu 14.04 system. The examples are known to be applicable to AIX, HPUX and Solaris.

To give you a specific use case wherein the below approach could be applicable had been the analysis of memory leaks in a program with a high amount of small memory allocations that in total however summed up to a high magnitude of gigabytes. Due to the high pressure on the dynamic memory allocator, approaches like compiler instrumentation (modern tools like the LLVM/Clang Leak Sanitizer had not been available) or in-depth program and heap analysis, e.g. by Valgrind, were not applicable due to speed issues. Our solution to address the problem was then to perform high-speed tracing of malloc(3), calloc(3), realloc(3) and free(3) functions and postpone the leak analysis to an off-line process running on the recorded runtime data.

As a first demonstration we try to intercept function calls to malloc(3) and free(3) and compile the two functions into a shared library called libintercept.so. To start we first lookup the function declarations of malloc(3) and free(3), for example by checking their respective man pages. The signature of both functions is:

Before we continue with the final coding of both functions, the most urgent question is how it is indeed possible to get our interception functions called instead of the real malloc(3) and free(3) functions in libc. Obviously, the whole code only works if our application is linked to libc (or whatever library we try to intercept) dynamically. If the program is linked statically, we cannot intercept the functions, though. Trying to keep the details of symbol resolution in dynamically linked applications at a minimum – please check this excellent post series for all the glory details – the dynamic loader decides at runtime which function to call by checking and matching the function symbols of all loaded shared libraries. If the same function symbol is exported by multiple loaded shared libraries, the order matters such that the first exported symbol is preferred. And this is exactly how we intercept the function calls by telling the dynamic loader to load our interception library first before the real library, in this case libc, is loaded.

Let’s get back to the code. This is how our interception functions are finally implemented:

The feature test macro _GNU_SOURCE is required to use the macro RTLD_NEXT. Next the header includes are defined whereof the header dlfcn.h is the one needed to interact with the dynamic loader as will be described later. The following lines of code define our version of the malloc function having the exact same signature as its original version. Inside the malloc function, we then firstly create a static function pointer for a function having malloc’s signature. The reason for using a static variable is that we do not want to query the address of the real malloc function for every function call to malloc so that this address is cached once “globally”. For the first entry into our malloc function we however have to ask the dynamic loader to find this address. This is accomplished by calling dlsym(3) with the special argument RTLD_NEXT. The functions from the dlfcn.h header file are in general used in Unix operating systems to load dynamic libraries and introspect the loaded dynamic libraries at runtime e.g. as opposed to at the application start-up time. The dlsym(3) may then be used in this context to find a symbol’s address in such a dynamic library. Using the special argument RTLD_NEXT we however instruct the dynamic loader to find the address of the next symbol in the search hierarchy having the name malloc – which is supposed to be our original version. While the first argument to dlsym(3) typically is a pointer to a loaded shared library, the special argument RTLD_NEXT refers to all loaded shared library, i.e. at application start-up and loaded by dlopen(3). Note that we would construct an endless-loop if RTLD_NEXT would not be used!

Once we found the address of the original malloc function, we perform our tracing and finally call the original function’s address retrieved before. That’s it! The code for free(3) is written likewise. A word of caution here about the tracing function: while functions from printf(3)-familiy may generally be used for tracing, they are critical for malloc(3). The reason is that printf(3), when used with format specifiers, internally calls malloc(3) so that you might end-up with an endless recursion crashing your stack.

Finally, how do we use the interception library to trace our application? As a highly simplified program we use the following main program to test our library:

Let’s compile and run the code:

If we execute the code, we do not get any output on the console, because the shared library libintercept.so is not loaded. This could be verified by running the ldd(1) utility to display dependent shared libraries:

To get our interception library loaded, we need to set the environment variable LD_PRELOAD to point to our library. In general, the environment variable LD_PRELOAD takes a colon- or space-separated list of libraries to be loaded before the dependent libraries are loaded. This could be verified by again using the ldd(1) utility:

To combine all of the above, we enable the tracing library for our application by the following invocation:

Variadic Functions

Almost all of the system calls and libc library functions have a fixed function signature with a pre-defined number of parameters. Exceptions are the functions from the printf(3)-family and the open(2) system call among others. For the printf(3) functions, the interception is easy because the libc library provides functions to pass in va_list(3)s, while for open(2) the specification is clear when the variadic parameter is used. For generic variadic functions, it is unfortunately not possible to intercept them unless the library provides a function taking a va_list as input argument.

Leave a Reply

Your email address will not be published. Required fields are marked *