[botan-devel] seg fault using rsa classes

Jack Lloyd lloyd at randombit.net
Fri Dec 6 16:28:07 EST 2013

On Tue, Dec 03, 2013 at 12:39:58PM -0800, Stuart Maclean wrote:
> Apologies for including code here, but I have a seg fault issue.  valgrind
> suggests some form of 'unpad' error.  This is on Linux (64bit), using Botan
> 1.10.6.  The new 1.10.6 api for doing rsa encryption and decryption uses the
> notion of PK_Encryptor I think, which is new to me.  Are there any docs
> describing their usage, other than doxygen??

There is some documentation in http://botan.randombit.net/pubkey.html as well.

> The seg fault below is during PK_Decryptor.decrypt().

Ouch! The problem is twofold - one was an integer underflow in a check
which allowed you to use a 512 bit key with EME1/SHA-256 even though
the key can't actually encode such a message (as OAEP adds a certain
amount of overhead and the result would be over 512 bits). The other
was a bug in decoding where if the input was so large it couldn't be
valid we would set the length to zero and then the code would go
reading an array off into the sunset until it hit an unmapped
page. Thanks for reporting this; can you try the following patch?

# patch "src/pk_pad/eme1/eme1.cpp"
#  from [6395b19097a454523f6a918d877db7b0f483f084]
#    to [1b0e06178127313bf96b78fa48e0d367e469fb9b]
--- src/pk_pad/eme1/eme1.cpp    6395b19097a454523f6a918d877db7b0f483f084
+++ src/pk_pad/eme1/eme1.cpp    1b0e06178127313bf96b78fa48e0d367e469fb9b
@@ -21,7 +21,7 @@ SecureVector<byte> EME1::pad(const byte 
    key_length /= 8;
-   if(in_length > key_length - 2*Phash.size() - 1)
+   if(key_length < in_length + 2*Phash.size() + 1)
       throw Invalid_Argument("EME1: Input is too large");
    SecureVector<byte> out(key_length);
@@ -82,7 +82,7 @@ SecureVector<byte> EME1::unpad(const byt
    * to timing analysis. Other compilers, or GCC on other platforms,
    * may or may not.
-   for(size_t i = delim_idx; i != input.size(); ++i)
+   for(size_t i = delim_idx; i < input.size(); ++i)
       const bool zero_p = !input[i];
       const bool one_p = input[i] == 0x01;

> One other point while I am on the topic.  I have always been confused as to
> why there is no method on RSA_PrivateKey for deriving the corresponding
> public key.

RSA_PrivateKey is a subclass of RSA_PublicKey so all that's needed is:

RSA_PublicKey pub = priv;

Or more likely in practice, calling X509::PEM_encode(priv), exporting
the string, and then X509::load_key in the receiving process.

Looking at it in hindsight an explicit public_key() method would much
cleaner than derivation here, but the nature of this project seems to
be design decisions I made when I was 20 continue to haunt me. :/


More information about the botan-devel mailing list