[Botan-devel] Re: crashing in dso

Mayur Patel patel at coredp.com
Thu Jan 31 12:38:29 EST 2008

botan-devel-request at randombit.net wrote:

>On Wed, Jan 30, 2008 at 04:41:11PM -0500, Mayur Patel wrote:
>>I've used the botan library (1.6.3) successfully in a couple simple 
>>executable binaries, using gcc 4.0.2 and Fedora Core 4.  But compiling 
>>the same code into dsos, I get crashes inside the load_key() function, 
>>during the destruction of a std::vector.
>By a DSO, you mean an .so that you load via dlopen/dlsym, or something
>else? I'm not sure if I understand exactly the situation here.
I think you understand the situation; I'm compiling to a .so -- shared 
object library.

I have a layer of code on top of Botan, which is used in several 
contexts. When we build binaries, we use static linking, so things are 
resolved at compile time with no ambiguity.

I statically link (Botan + our layer) into a static library (.a) - this 
library is used to compile either binary executables or .so shared 
object libraries for 3rd party applications (plugins).

I tested pretty extensively compiling executable binaries and had no 
issues at all; so I deployed. But since I began using it in plugins, 
it's kicked up this issue not previously observed.

>>I'm confident that I've isolated the problem to be somewhere between the 
>>std::allocator and the botan library.  I am still able to get it to 
>>crash by purposely leaking memory - by not deleting the returned memory 
>>from load_key().  In that case, it's crashing in the std::allocator 
>>later in the software, during a std::vector allocation; so I'm very 
>>suspicious of interactions between the std::allocator and mechanisms in 
>Yikes. It is possible there is some corruption occuring here that is
>affecting internal std::allocator state (though I feel that is
>reasonably unlikely), but certainly Botan does not directly mess with
>the std allocators or containers.
>I suspect this may be related to the problem described in:
>   http://gcc.gnu.org/faq.html#dso

Yikes, indeed. I was unaware of this issue with gcc; however since we're 
writing plugins for a commercial app it's not easy to determine from our 
side whether they're opening the .so in the "approved way." The compiler 
version used for these particular plugins is gcc 4.0.2.

I'll investigate further; this is a very good lead. Thank you. I am 
using dynamic_cast on the return value of load_key().

>My best guess right now is that due to the DSO usage, two copies of
>libstdc++ are being loaded, and if/when a pointer is allocated with
>one and freed to another: boom.

It's not clear to me how that situation could occur. I don't seem to be 
getting crashes using normal new[] and delete[] which is used in the 
code; specifically seems to be in std::allocators; but I will not rule 
anything out right now.

>It would help if you could post the code as well as the exact invocation
>syntax you're using with GCC to build and link everything.
I'll investigate further, and if necessary I'll try to put together the 
repro you describe. It's embedded a bit, so will take a little effort to 
strip down....

Thanks for the leads; I will reply once I have additional information.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: patel.vcf
Type: text/x-vcard
Size: 145 bytes
Desc: not available
URL: <http://lists.randombit.net/pipermail/botan-devel/attachments/20080131/10ea9689/attachment.vcf>

More information about the botan-devel mailing list