[Botan-devel] Re: [Monotone-devel] oprofile data for mtn 0.37.

[Jack Lloyd]

On Thu, Mar 13, 2008 at 12:07:57PM -0400, Zack Weinberg wrote:
> On Thu, Mar 13, 2008 at 11:08 AM, Jack Lloyd <lloyd at randombit.net> wrote:
> >  Pipe actually keeps each message around forever (all storage is
> >  reclaimed once a) all output has been read out b) the message in
> >  question is not currently being processed). You can read out a out of
> >  order / interleaved with reading others / whatever.
> 
> Oh, I see now.
> 
> Is *all* storage reclaimed, or is there some array indexed by message
> number that's going to keep growing forever if I recycle pipes?

It should be everything (ie, zero increased memory usage over time, as
long as you read the entire contents of each message out) - the code
handling this is in out_buf.cpp if you would like to review.

Message numbers are currently stored as 32-bit integers, and there
will probably be failures on overflow (in particular at 2^32-2 ==
Pipe::LAST_MESSAGE) -- how likely would a process-persistent Pipe
eventually hit this limit in (say) long-running a monontone server?

You have: 2^32 / (300 / s)
You want: day
        * 165.7009

The message number can (and probably should) be changed to a long
long, though also the underlying issue that Pipe should detect/signal
this condition needs fixing.

WRT leaks, attached is a simple testcase, which I have only run up to
about a million messages (slow machine), but there was no perceptible
increase in memory usage as the # of messages run through the Pipe
increases. I will attempt running this out to 2^32 messages to make
sure though (and confirm/deny if there are failures at overflow).

-Jack
-------------- next part --------------
#include <botan/filters.h>
#include <botan/pipe.h>
#include <botan/loadstor.h>

#include <unistd.h>

using namespace Botan;

int main()
   {
   Pipe p;

   for(u32bit j = 0; j != 10000000; j++)
      {
      if(j % (1024*1024) == 0)
         printf("%d\n", j);

      p.start_msg();
      for(u32bit k = 0; k != 4; k++)
         p.write(get_byte(k, j));

      p.end_msg();

      SecureVector<byte> m = p.read_all(Pipe::LAST_MESSAGE);

      if(m.size() != 4)
         printf("Bad size %d at %d\n", m.size(), j);

      for(u32bit k = 0; k != 4; k++)
         if(m[k] != get_byte(k, j))
            printf("Bad output for %d, byte %d: %02X != %02X",
                   j, k, m[k], get_byte(k, j));
      }

   printf("done\n");

   pause();
   }