[Botan-devel] clearing pipe messages

Jack Lloyd lloyd at randombit.net
Mon Dec 19 15:31:47 EST 2005

1.4.10 has changes which should resolve this problem - it works automatically
and is (or should be) completely backwards compatible.

Essentially, Pipe (or, actually, the new Output_Buffer class) now mantains a
std::deque of SecureQueue*. Pipe calls Output_Buffer::retire() as part of
end_msg(), at which point the Output_Buffer will delete any old SecureQueue*s
which have no output saved in them. The Output_Buffer will remove from the head
of the queue any deleted queues, and keep track of how many it has deleted
(this way, message numbers don't change; meaning you still run into trouble
after 4 billion messages right now, but support 64-bit message numbers raises
complications, and really doesn't seem that worthwhile IMO). So you never take
the overhead of an empty SecureQueue; they are deleted as soon as soon as we
can tell that they will not be used further. You do take overhead for the
std::deque if you leave data in the Pipe, for example if you have

msg=1: some data
msg=2...N: empty

this will cause N slots to be used by the deque, but as soon as you read the
entire contents of msg=1 out of the pipe, Output_Buffer will see that the first
N messages are all deleted and remove them all from the queue. The empty
messages are actually NULL pointers (at least, they are once retire() is called
and empty ones are deleted); Output_Buffer treats a NULL SecureQueue* as one
which has no data stored in it.

Using std::map or hash_map would remove this worst-case problem, but it seemed
to cause a lot more overhead than it is worth in typical cases, and by far the
common case is that each message is read out entirely in a fairly short amount
of time. This method didn't seem doesn't seem to cause any major overhead, and
handles this case very well. The refactoring of this queue managment code out
to another class means it will be easier to replace for those with specialized
requirements, as well.


On Wed, Dec 14, 2005 at 11:33:36PM +0500, Muzaffar Mahkamov wrote:
> Jack,
> Reusing the pipe helps a bit. Profiler shows 1.5-2 times performance
> gain in my case (I call the Encrypt() function a lot, several thousand
> times). But in the long run it'll introduce the problems you've mentioned.
> I think it'd be great to have a function in Pipe that clears the
> 0..message_count()-1 messages alone and resets the current message
> number without touching the filters (destruct() method destroys both
> messages and filters) and unread messages. Thus the
> library would be compatible with older versions and older
> applications. New applications would explicitly call the function if
> they need it. If you think it's possible just by clearing the message
> vector I could do and test that on my side.
> Muzaffar

More information about the botan-devel mailing list