[Botan-devel] Small file ciphering speed

Jack Lloyd lloyd at randombit.net
Thu Sep 25 13:20:01 EDT 2008

On Thu, Sep 25, 2008 at 04:02:36PM +0800, Mr Diggilin wrote:
> This is all very good news. Those timings are particularly nice, I'm
> looking forward to testing an implementation with the database.
> Unfortunately, I wasn't able to compile your example as given, since I
> didn't have a "botan/loadstore.h". I'm using botan 1.6.4 from the
> openSUSE repositories, so perhaps that's not available till a later
> version?

Oops, yes, I wrote that code against mainline (1.7) some small changes
are needed for 1.6 - I've attached a version for that.

> I couldn't immediately find a good way to set the salt without that
> store_le function, so I just hardcoded the salt "salt=1" (doing so in
> the loop made the execution time an order of magnitude slower, so I just
> placed it outside).

I wrote code that instead emulates the store* functions and writes the
4 bytes of the loop index i to salt in big-endian form.

> The new_random_salt had only one "version" taking a
> u32bit for how many bytes so I just removed the rng argument.

Yes in Botan 1.6 there is only the single global RNG (which
new_random_salt and other functions get an implicit reference to via
the global library state described in libstate.h).

> at 1.863 seconds. I'm not sure how much of a difference the salt setting
> takes, but it may add some time if that isn't hardcoded.

Since EAX does not require random nonces, you can use things like loop
indexes or db row ids directly without concern (as mentioned
previously, as long as they never dup).

> In general I can get 3 mbps on a large (200mb) file with Twofish/EAX
> (~2 times that using CBC without auth), which seems very low,
> indicating something else may not be doing well on this machine in
> general.

Yikes. The 2x slowdown with EAX vs CBC strongly suggests it is CPU
limited, which really surprises me since I would have expected that
disk latency would be the problem once that size of file is reached.

FWIW, with a randomly generated 200 mb input file:

$ dd if=/dev/urandom bs=1M count=200  of=input
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 37.9952 s, 5.5 MB/s

And this test prog:

#include <botan/botan.h>
#include <botan/eax.h>

#include <fstream>

using namespace Botan;

int main()
   std::ifstream in("input");

   LibraryInitializer init;

   SymmetricKey key(32);
   SymmetricKey iv(16);

   Pipe pipe(new EAX_Encryption("Twofish", key, iv),
             new DataSink_Stream("output"));

   in >> pipe;

With 1.6.4 does 26.5 Mb/sec sustained on my Core2 (and that is whole
program runtime, so includes at least some of the disk transfer time,
etc, though this box has plenty of mem).

> Note that that speed
> is measuring nothing but start_msg() to end_msg().
> As for raw cipher speed not including initialization in this case...
> Encryption: 671 ms
> Decrypt/compare: 701 ms
> A bit of calculation guesstimates that at a bit less than 1mbps.


Are you buffering inputs to Pipe? Byte-at-a-time vs page-at-a-time can
easily cause an order of magnitude difference.

Also: is this processor 32 or 64 bit? Which version of GCC are you
using? Can you send the contents of /proc/cpuinfo?

I am almost tempted to suggest building Botan from source and seeing
if that makes any difference. I have not really looked at how the SuSE
packaging was done (actually in the last year I wasn't even entirely
sure that SuSE was shipping Botan as I can't find anything on the
site), but I know the packaging was done by a SuSE engineer for
commercial customers who wanted it, and he contacted me a few times
with questions about things and seemed to get it, so I would have
thought everything would be fine, but now I'm slightly worried that
the openSuSE package might just be badly miscompiled somehow.

> I'm hoping to get it working in the program soon to see what kind of
> speeds I'll get in real life. Is there a good way of putting an int in a
> MemoryVector while lacking store_le?

Check out attached version of row_encryptor for 1.6.x

> Thank you very much for all your time and how helpful you've been, I'm
> truly thankful.

Well it is not completely altruistic, I like seeing Botan be used in
interesting projects and while I am not really sure what you are
working on you are asking interesting questions at least so hopefully
it is something cool. ;)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: row_encryptor.cpp
Type: text/x-c++src
Size: 3421 bytes
Desc: not available
URL: <http://lists.randombit.net/pipermail/botan-devel/attachments/20080925/e5e9827a/attachment.bin>

More information about the botan-devel mailing list