This is a translation (with some additions) of the articles I have written in 2012 in the old blog which does not exist anymore.

Originally I faced this issue in Qt when for some reason one of event handlers had thrown bad_alloc() exception. That was quite a large application, and it was not easy to understand which one and why that happened.

In Qt, if an application throws an exception which is never handled by the application, the exception lands to the Qt event loop, and you get a message like this:

Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt. You must
reimplement QApplication::notify() and catch all exceptions there.

Then Qt rethrows the exception, and the application is usually terminated. With libstdc++ you may see some additional diagnostics like this:

terminate called after throwing an instance of ‘std::bad_alloc’
what(): std::bad_alloc

The issue is that you cannot get the backtrace from the uncaught exception handler: the backtrace will point you to the place from which the exception was rethrown, that is, into the Qt event loop.

To get the correct backtrace, you need to understand how C++ throws exceptions. The specification is available in C++ ABI for Itanium. In particular, you need Section 2.4 Throwing an Exception:

In broad outline, a possible implementation of the processing necessary to throw an exception includes the following steps:

  • Call <strong>__cxa_allocate_exception</strong> to create an exception object (see Section 2.4.2).
  • Evaluate the thrown expression, and copy it into the buffer returned by <strong>__cxa_allocate_exception</strong>, possibly using a copy constructor. If evaluation of the thrown expression exits by throwing an exception, that exception will propagate instead of the expression itself. Cleanup code must ensure that <strong>__cxa_free_exception</strong> is called on the just allocated exception object. (If the copy constructor itself exits by throwing an exception, <strong>terminate()</strong> is called.)
  • Call <strong>__cxa_throw</strong> to pass the exception to the runtime library (see Section 2.4.3). <strong>__cxa_throw</strong> never returns.

One of the possible solutions is to use gdb: you set a breakpoint on __cxa_throw (b __cxa_throw), run the program, and when the breakpoint is reached, you use bt to get the backtrace.

Something like this:

$ gdb buggy_app 
GNU gdb (Ubuntu/Linaro 7.4-2012.02-0ubuntu2) 7.4-2012.02
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /path/to/buggy_app...done.
(gdb) start
Temporary breakpoint 1 at 0x405f6d: file main.cpp, line 75.
Starting program: /path/to/buggy_app
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=3, argv=0x7fffffffe008) at main.cpp:75
75              QCoreApplication a(argc, argv);
(gdb) b __cxa_throw
Breakpoint 2 at 0x7ffff7154910
(gdb) c
Continuing.

Breakpoint 2, 0x00007ffff7154910 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff7154910 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff762d102 in qBadAlloc () at global/qglobal.cpp:1994
#2  0x00007ffff7fe6e7e in DnsRequestQueuePrivate::_q_resultsReady (this=0x43af90, r=..., id=11771, code=0, ctx=...) at dnsrequestqueue.cpp:83
#3  0x00007ffff7fe7629 in DnsRequestQueue::qt_static_metacall (_o=0x7fffffffded8, _c=QMetaObject::InvokeMetaMethod, _id=1, _a=0x43d130) at debug/moc_dnsrequestqueue.cpp:53
#4  0x00007ffff7750446 in QObject::event (this=0x7fffffffded8, e=) at kernel/qobject.cpp:1195
#5  0x00007ffff7736e9c in QCoreApplication::notifyInternal (this=0x7fffffffdf00, receiver=0x7fffffffded8, event=0x43d310) at kernel/qcoreapplication.cpp:876
...

However, this is not a pleasant task, especially if the application is multi-threaded or if it uses exceptions for error handling. In this case you may want to automate the process.

So, get back to the C++ ABI. We need Sections 2.4.3 Throwing the Exception Object and 2.5.4 Rethrowing Exceptions. Below are the prototypes for __cxa_throw() (throws an exceptions) and __cxa_rethrow() (rethrows the current exception) functions:

void __cxa_throw (void *thrown_exception, std::type_info *tinfo, void (*dest) (void *) );
void __cxa_rethrow ();

  • thrown_exception is the address of the thrown exception object (which points to the throw value, after the header — see Data Structures for the layout);
  • tinfo gives the static type of the throw argument as a std::type_info pointer, used for matching potential catch sites to the thrown exception;
  • dest is the destructor pointer to be used eventually to destroy the object.

Case 1: Throw Exception

The only thing we need here is tinfo: tinfo->name() will give us the name of the type of the thrown exception. However, the name of the type will look like St13runtime_error (the so-called mangled name). To get the human-readable name, we can use Demangler API:

namespace abi {
  extern "C" char* __cxa_demangle (const char* mangled_name,
				   char* buf,
				   size_t* n,
				   int* status);
}

Sample code:

char* demangled = abi:: __cxa_demangle(tinfo->name(), nullptr, nullptr, nullptr);
std::cerr << "Thrown exception of type " << (demangled ? demangled : tinfo->name()) << std::endl;
if (demangled) {
    std::free(demangled);
}

We can check here whether the thrown exception is inherited from std::exception, and if so, we can print exception::what(). The issue here is that thrown_exception is of type void* (in C++, you can throw anything, not necessarily objects), and as a consequence, dynamic_cast will not work. To understand the magic behind dynamic_cast we need to read the ABI document again: this time, we need Run-Time Type Information.

Dynamic casts are implemented by __dynamic_cast() function:

   extern "C" 
   void* __dynamic_cast ( const void *sub,
			  const abi::__class_type_info *src,
			  const abi::__class_type_info *dst,
			  std::ptrdiff_t src2dst_offset);
   /* sub: source address to be adjusted; nonnull, and since the
    *      source object is polymorphic, *(void**)sub is a virtual
    pointer.
    * src: static type of the source object.
    * dst: destination type (the "T" in "dynamic_cast<T>(v)").
    * src2dst_offset: a static hint about the location of the
    *    source subobject with respect to the complete object;
    *    special negative values are:
    *       -1: no hint
    *       -2: src is not a public base of dst
    *       -3: src is a multiple public base type but never a
    *           virtual base type
    *    otherwise, the src type is a unique public nonvirtual
    *    base type of dst at offset src2dst_offset from the
    *    origin of dst.
    */

The algorithm is as follows:

  1. We need to check if tinfo is an instance of abi::__class_type_info (see Section 2.9.5 RTTI Layout for details); if it isn’t, then the thrown exception is not an object, and there is nothing to do here.
  2. Cast &typeid(std::exception) to const abi::__class_type_info*.
  3. Call __dynamic_cast and check whether thrown_exception is inherited from std::exception.

const abi::__class_type_info* exc = dynamic_cast<const abi::__class_type_info*>(&typeid(std::exception));
const abi::__class_type_info* cti = dynamic_cast<abi::__class_type_info*>(tinfo);
if (cti && exc) {
    std::exception* the_exception = reinterpret_cast<std::exception*>(abi::__dynamic_cast(thrown_exception, exc, cti, -1));
    if (the_exception) {
        std::cout << the_exception->what() << std::endl;
    }
}

All that is left is to get the backtrace. There is nothing difficult here though.

Case 2: Rethrow Exception

__cxa_rethrow() is invoked from a catch block and re-throws the caught exception. The function has no arguments, and therefore we need to get somehow the rethrown exception. If we read Section 2.2.2 Caught Exception Stack, we will find out that we can use the following function:

__cxa_eh_globals *__cxa_get_globals(void);

struct __cxa_eh_globals {
	__cxa_exception *	caughtExceptions;
	unsigned int		uncaughtExceptions;
};

__cxa_exception is defined in Section 2.2.1 C++ Exception Objects:

struct __cxa_exception {
	std::type_info* exceptionType;
	void (*exceptionDestructor)(void*);
	std::unexpected_handler unexpectedHandler;
	std::terminate_handler terminateHandler;
	__cxa_exception* nextException;
	int handlerCount;
	int handlerSwitchValue;
	const char* actionRecord;
	const char* languageSpecificData;
	void* catchTemp;
	void* adjustedPtr;
	_Unwind_Exception unwindHeader;
};

_Unwind_Exception is defined in Section 1.2 Data Structures and is available in <unwind.h> system header.

__cxa_exception itself is the header; the thrown exception object follows it.

__cxa_eh_globals* g = __cxa_get_globals();
if (g && g->caughtExceptions) {
    void* thrown_exception = reinterpret_cast<uint8_t*>(g->caughtExceptions) + sizeof(struct __cxa_exception);
    std::type_info* tinfo  = g->caughtExceptions->exceptionType;

    // The rest is the same as in __cxa_throw()
}

Done. Now we need to intercept __cxa_throw and __cxa_rethrow. This shlild not be difficult:

#include <dlfcn.h>

typedef void(*cxa_throw_type)(void*, std::type_info*, void(*)(void*));
typedef void(*cxa_rethrow_type)(void);

cxa_throw_type   orig_cxa_throw   = reinterpret_cast<cxa_throw_type>(dlsym(RTLD_NEXT, "__cxa_throw"));
cxa_rethrow_type orig_cxa_rethrow = reinterpret_cast<cxa_rethrow_type>(dlsym(RTLD_NEXT, "__cxa_rethrow"));

Putting it all together:

#include <typeinfo>
#include <exception>
#include <dlfcn.h>
#include <pthread.h>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <stdexcept>
#include <execinfo.h>
#include <cxxabi.h>
#include <unwind.h>
#include <unistd.h>

struct __cxa_exception {
	std::type_info* exceptionType;
	void (*exceptionDestructor)(void*);
	std::unexpected_handler unexpectedHandler;
	std::terminate_handler terminateHandler;
	__cxa_exception* nextException;
	int handlerCount;
	int handlerSwitchValue;
	const char* actionRecord;
	const char* languageSpecificData;
	void* catchTemp;
	void* adjustedPtr;
	_Unwind_Exception unwindHeader;
};

struct __cxa_eh_globals {
	__cxa_exception* caughtExceptions;
	unsigned int uncaughtExceptions;
};

extern "C" __cxa_eh_globals* __cxa_get_globals(void);

using cxa_throw_type   = void(*)(void*, std::type_info*, void(*)(void*));
using cxa_rethrow_type = void(*)();

static cxa_throw_type   orig_cxa_throw   = nullptr; // Address of the original __cxa_throw
static cxa_rethrow_type orig_cxa_rethrow = nullptr; // Address of the original __cxa_rethrow

/**
 * Get the backtrace.
 * Here and below we use functions from the C library; these functions do not throw exceptions.
 */
static void get_backtrace()
{
	static void* buf[128];

	int n = backtrace(buf, 128);
	std::fprintf(stderr, "%s\n", "*** BACKTRACE ***");
	backtrace_symbols_fd(buf, n, STDERR_FILENO);
}

/**
 * Exception handling common for both __cxa_throw and __cxa_rethrow
 */
static void handle_exception(void* thrown_exception, std::type_info* tinfo, bool rethrown)
{
	char* demangled = abi:: __cxa_demangle(tinfo->name(), 0, 0, 0);
	std::fprintf(stderr, "%s exception of type %s\n", (rethrown ? "Rethrown" : "Thrown"), (demangled ? demangled : tinfo->name()));
	if (demangled) {
		std::free(demangled);
	}

	const abi::__class_type_info* exc = dynamic_cast<const abi::__class_type_info*>(&typeid(std::exception));
	const abi::__class_type_info* cti = dynamic_cast<abi::__class_type_info*>(tinfo);

	if (cti && exc) {
		std::exception* the_exception = reinterpret_cast<std::exception*>(abi::__dynamic_cast(thrown_exception, exc, cti, -1));
		if (the_exception) {
			std::fprintf(stderr, "what(): %s\n", the_exception->what());
		}
	}

	get_backtrace();
	std::fprintf(stderr, "\n\n");
}

// The functions below should go to an anonymous namespace, otherwise g++ becomes crazy and complains about
// mismatched types in throw statements
namespace {

extern "C" void __cxa_throw(void* thrown_exception, std::type_info* tinfo, void (*dest)(void*))
{
	handle_exception(thrown_exception, tinfo, false);
	
	if (orig_cxa_throw) {
		orig_cxa_throw(thrown_exception, tinfo, dest);
	}
	else {
		std::terminate();
	}
}

extern "C" void __cxa_rethrow(void)
{
	__cxa_eh_globals* g = __cxa_get_globals();
	if (g && g->caughtExceptions) {
		void* thrown_exception = reinterpret_cast<uint8_t*>(g->caughtExceptions) + sizeof(struct __cxa_exception);
		handle_exception(thrown_exception, g->caughtExceptions->exceptionType, true);
	}

	if (orig_cxa_rethrow) {
		orig_cxa_rethrow();
	}
	else {
		std::terminate();
	}
}

}

/**
 * Initialization. This can probably be done from the exception handler.
 */
static void initialize()
{
	orig_cxa_throw   = reinterpret_cast<cxa_throw_type>(dlsym(RTLD_NEXT, "__cxa_throw"));
	orig_cxa_rethrow = reinterpret_cast<cxa_rethrow_type>(dlsym(RTLD_NEXT, "__cxa_rethrow"));
}

int main(int, char**)
{
	initialize();

	try {
		try {
			throw std::runtime_error("123");
		}
		catch (const std::exception& e) {
			std::printf("e.what(): %s\n", e.what());
			throw;
		}
	}
	catch (const std::exception& d) {
		std::printf("d.what(): %s\n", d.what());
	}

	try {
		throw 1;
	}
	catch (int x) {
		std::printf("%d\n", x);
	}
	
	return 0;
}

Sample output:

$ g++ test.cpp -O2 -g -o test -ldl
$ ./test
Thrown exception of type std::runtime_error
what(): 123
*** BACKTRACE ***
./test(+0xf13)[0x563157016f13]
./test(+0x10b0)[0x5631570170b0]
./test(__cxa_throw+0x2c)[0x5631570170ff]
./test(+0x1224)[0x563157017224]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0bf8b831c1]
./test(+0xe1a)[0x563157016e1a]


e.what(): 123
Rethrown exception of type std::runtime_error
what(): 123
*** BACKTRACE ***
./test(+0xf13)[0x563157016f13]
./test(+0x10b0)[0x5631570170b0]
./test(__cxa_rethrow+0x51)[0x56315701717d]
./test(+0x1281)[0x563157017281]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0bf8b831c1]
./test(+0xe1a)[0x563157016e1a]


d.what(): 123
Thrown exception of type int
*** BACKTRACE ***
./test(+0xf13)[0x563157016f13]
./test(+0x10b0)[0x5631570170b0]
./test(__cxa_throw+0x2c)[0x5631570170ff]
./test(+0x1300)[0x563157017300]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0bf8b831c1]
./test(+0xe1a)[0x563157016e1a]


1

How to Get Line Numbers

Depending on how the executable was built, it may be very difficult to get source file and line number information from addresses in the backtrace.

For example, the code above was built with -pie (meaning it is position independent executable), and addr2line will not work as expected.

Position Independent Executables

If Address Space Layout Randomization is on (can be checked with sysctl kernel.randomize_va_space; if the value is non-zero, it is enabled), it needs to be disabled. Instead of disabling it globally, we can use setarch to disable it just for our binary:

setarch $(uname -m) -R ./test

Now, the executable will always be loaded at the same address. However, that address is not within the executable, and for addr2line to work, addresses need to be recalculated:

setarch {{EJS43}} -R sh -c 'LD_TRACE_PRELINKING=1 ./test | grep "=>"'

This will show something like:

./test => ./test (0x0000555555554000, 0x0000555555554000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffff7bd1000, 0x00007ffff7bd1000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffff784b000, 0x00007ffff784b000) TLS(0x1, 0x0000000000000020)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffff7634000, 0x00007ffff7634000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff7254000, 0x00007ffff7254000) TLS(0x2, 0x00000000000000b0)
/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007ffff7dd5000, 0x00007ffff7dd5000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffff6efe000, 0x00007ffff6efe000)

In the example above, 0x0000555555554000 is the offset we need to subtract from the address.

Thus, if we get something like this from our program:

./test(+0x1033)[0x555555555033]
./test(+0x1382)[0x555555555382]
./test(__cxa_throw+0x2c)[0x5555555553d1]
./test(+0x15f9)[0x5555555555f9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7ffff72751c1]
./test(+0xf3a)[0x555555554f3a]

We need to feed 0x1033, 0x1382, 0x13d1, 0x15f9, 0xf3a to addr2line (if you look carefully, these are the addresses shown in parentheses; if your backtrace always lands to main(), you can calculate the amendment yourself by subtracting the short address from the long).

UPDATE: more information is available here.

Normal Executables

For non-PIE addresses shown by backtrace_symbols_fd() can be fed directly to addr2line.

Automated Solution that Works for Both Types

Calculating offsets and invoking addr2line manually can be tedious and of course needs to be automated. If you don’t want or cannot rebuild the executable with -no-pie, I still have a solution.

First, we need to install eu-addr2line program (from elfutils; in Debian-based systems this is as easy as sudo apt install elfutils).

Then we need to add some code to our program.

It will need one extra header:

#include <limits>

and some code added to static void get_backtrace() right after backtrace_symbols_fd(buf, n, STDERR_FILENO);:

	std::size_t bufsize = (2*sizeof(void*) + 2 /* 0x */ + 1 /* space */)*n + std::strlen("/usr/bin/eu-addr2line --pretty-print -ifCa -p  1>&2") + (std::numeric_limits<pid_t>::digits10 + 1) + 1;
	char* space = reinterpret_cast<char*>(std::calloc(bufsize, 1));
	if (space) {
		char* orig = space;
		int c = std::sprintf(space, "/usr/bin/eu-addr2line --pretty-print -ifCa -p %d ", getpid());
		space += c;
		for (int i=0; i<n; ++i) {
			c = std::sprintf(space, "%p ", buf[i]);
			space += c;
		}

		std::sprintf(space, "%s", "1>&2");
		std::fprintf(stderr, "%s\n", "\n*** DECODED BACKTRACE ***");

		std::fprintf(stderr, "%s\n", orig);
		if (std::system(orig)) {}
		std::free(orig);
	}

Let’s see what is going on here.

In line 1 we calculate the space needed for the command line:

  • (2*sizeof(void*) + 2 /* 0x */ + 1 /* space */) is the maximum size of a single hexadecimal address (void*); sizeof(void*) is multiplied by two to get the number of hexadecimal digits (1 byte = 2 digits), and added two bytes for 0x prefix, and one more byte for the space after the address;
  • the result is multiplied by n (number of entries in the backtrace);
  • the length of the fixed portion of the command line is added;
  • (std::numeric_limits<pid_t>::digits10 + 1) is the number of digits needed to display a value of pid_t;
  • finally, add one to account for the terminating zero.

Lines 5—12 generate the command line, line 15 prints the command to be run, line 16 executes it (if is added to suppress gcc’s warning about the not checked return code), and line 17 releases the memory we have allocated.

The output of the program will now look like this:

Thrown exception of type std::runtime_error
what(): 123
*** BACKTRACE ***
./test(+0x1033)[0x55bb3b305033]
./test(+0x12fb)[0x55bb3b3052fb]
./test(__cxa_throw+0x2c)[0x55bb3b30534a]
./test(+0x146f)[0x55bb3b30546f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fcfd58811c1]
./test(+0xf3a)[0x55bb3b304f3a]

*** DECODED BACKTRACE ***
/usr/bin/eu-addr2line --pretty-print -ifCa -p 6941 0x55bb3b305033 0x55bb3b3052fb 0x55bb3b30534a 0x55bb3b30546f 0x7fcfd58811c1 0x55bb3b304f3a 1>&2
get_backtrace at /tmp/test.cpp:51
handle_exception at /tmp/test.cpp:97
__cxa_throw at /tmp/test.cpp:108
main at /tmp/test.cpp:149
__libc_start_main at ../csu/libc-start.c:342
_start at ??:0


e.what(): 123
Rethrown exception of type std::runtime_error
what(): 123
*** BACKTRACE ***
./test(+0x1033)[0x55bb3b305033]
./test(+0x12fb)[0x55bb3b3052fb]
./test(__cxa_rethrow+0x51)[0x55bb3b3053c8]
./test(+0x14cc)[0x55bb3b3054cc]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fcfd58811c1]
./test(+0xf3a)[0x55bb3b304f3a]

*** DECODED BACKTRACE ***
/usr/bin/eu-addr2line --pretty-print -ifCa -p 6941 0x55bb3b305033 0x55bb3b3052fb 0x55bb3b3053c8 0x55bb3b3054cc 0x7fcfd58811c1 0x55bb3b304f3a 1>&2
get_backtrace at /tmp/test.cpp:51
handle_exception at /tmp/test.cpp:97
__cxa_rethrow at /tmp/test.cpp:124
main at /tmp/test.cpp:153
__libc_start_main at ../csu/libc-start.c:342
_start at ??:0


d.what(): 123
Thrown exception of type int
*** BACKTRACE ***
./test(+0x1033)[0x55bb3b305033]
./test(+0x12fb)[0x55bb3b3052fb]
./test(__cxa_throw+0x2c)[0x55bb3b30534a]
./test(+0x154b)[0x55bb3b30554b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fcfd58811c1]
./test(+0xf3a)[0x55bb3b304f3a]

*** DECODED BACKTRACE ***
/usr/bin/eu-addr2line --pretty-print -ifCa -p 6941 0x55bb3b305033 0x55bb3b3052fb 0x55bb3b30534a 0x55bb3b30554b 0x7fcfd58811c1 0x55bb3b304f3a 1>&2
get_backtrace at /tmp/test.cpp:51
handle_exception at /tmp/test.cpp:97
__cxa_throw at /tmp/test.cpp:108
main at /tmp/test.cpp:161
__libc_start_main at ../csu/libc-start.c:342
_start at ??:0


1
How to Get the Source of an Uncaught Exception in C++
Tagged on:         

Leave a Reply

Your email address will not be published.