[Botan-devel] crashing in dso

[Jack Lloyd]

On Wed, Jan 30, 2008 at 04:41:11PM -0500, Mayur Patel wrote:

Hi Mayur,

> I've used the botan library (1.6.3) successfully in a couple simple 
> executable binaries, using gcc 4.0.2 and Fedora Core 4.  But compiling 
> the same code into dsos, I get crashes inside the load_key() function, 
> during the destruction of a std::vector.

By a DSO, you mean an .so that you load via dlopen/dlsym, or something
else? I'm not sure if I understand exactly the situation here.

> I'm confident that I've isolated the problem to be somewhere between the 
> std::allocator and the botan library.  I am still able to get it to 
> crash by purposely leaking memory - by not deleting the returned memory 
> from load_key().  In that case, it's crashing in the std::allocator 
> later in the software, during a std::vector allocation; so I'm very 
> suspicious of interactions between the std::allocator and mechanisms in 
> botan.

Yikes. It is possible there is some corruption occuring here that is
affecting internal std::allocator state (though I feel that is
reasonably unlikely), but certainly Botan does not directly mess with
the std allocators or containers.

I suspect this may be related to the problem described in:
   http://gcc.gnu.org/faq.html#dso

My best guess right now is that due to the DSO usage, two copies of
libstdc++ are being loaded, and if/when a pointer is allocated with
one and freed to another: boom.

It would help if you could post the code as well as the exact invocation
syntax you're using with GCC to build and link everything.

-Jack