[Botan-devel] clearing pipe messages

Muzaffar Mahkamov mahkamov at gmail.com
Wed Dec 14 13:33:36 EST 2005


Jack,

Reusing the pipe helps a bit. Profiler shows 1.5-2 times performance
gain in my case (I call the Encrypt() function a lot, several thousand
times). But in the long run it'll introduce the problems you've mentioned.

I think it'd be great to have a function in Pipe that clears the
0..message_count()-1 messages alone and resets the current message
number without touching the filters (destruct() method destroys both
messages and filters) and unread messages. Thus the
library would be compatible with older versions and older
applications. New applications would explicitly call the function if
they need it. If you think it's possible just by clearing the message
vector I could do and test that on my side.

Muzaffar


Wednesday, December 14, 2005, 9:21:25 PM, you wrote:

> You're exactly right here; if you're handling multiple messages through a
> single Pipe, you have to handle keeping track of the message number you're on
> (which is typically as easy as calling pipe.message_count() and subtracting
> one)

> Currently MAX_MESSAGES is defined as as 0xFFFFFFF0; on most current machines,
> Pipe will give you serious problems long before that; just holding the (empty)
> message queues for that many messages will take at least 32 gigabytes, and that
> is not considering the overhead of std::vector (which is probably fairly
> minimal, but still nonzero).

> There is no way to reset the message number; the best way to handle this
> without modifying Botan would be to have a wrapper class that periodically
> (say, after 2^16 messages) destroys and reallocates the Pipe object it is
> holding, and otherwise just forwards everything to the Pipe. The most obvious
> place where you could insert code that deleted the messages and reset the
> message counter would be Pipe::destroy, if you wanted to carry a modified
> version.

> Looking at it, maintaining these structures seems like needless overhead, and
> removing it may end up being an interesting problem. I have a couple of ideas
> that I'll try out tonight, so this problem may be removed in the near
> future. It's easy to simply destroy the messages, but it seems a little
> trickier to also get

> automatic operation - all existing application code benefits from the change

> -and-

> compatibility - current application code behaves as expected (there are some
>                 nice corner case in here, so it's not as easy as it seems)

> both of which seem like necessary parts of the problem. Once the scaling issue
> is solved (which I think it will - I have two different methods in mind right
> now that would _work_, I'm just not sure how clean they will be to implement or
> how fast they will be (yet)), that does still leave the original problem with
> MAX_MESSAGES; however, most of the Pipe code is structured so that it would be
> pretty easy to change the message number to a 64-bit value. Right now there is
> no point in doing so, because the aformentioned overhead will drop your process
> long before you hit 2^32 messages.

> -Jack

> On Wed, Dec 14, 2005 at 04:22:43PM +0500, Muzaffar Mahkamov wrote:
>> Hi Jack,
>> 
>> I think I've found the problem. read() operation increments the
>> message number and the next time i was attempting to read the default
>> message (which is empty) and the new message remained in the pipe. If
>> I increment the message number and pass it to read() operation
>> everything works fine. On the other hand, message number may reach
>> Botan::Pipe::MAX_MESSAGES in my case. Is there a function to reset the
>> message number or how could I do that without side effects by tweaking
>> the Botan code? 
>> 
>> Muzaffar
>> 
> _______________________________________________
> botan-devel mailing list
> botan-devel at randombit.net
> http://www.randombit.net/mailman/listinfo/botan-devel


-- 
Best regards,
 Muzaffar                            mailto:mahkamov at gmail.com




More information about the botan-devel mailing list