[cryptography] rolling hashes, EDC/ECC vs MAC/MIC, etc.

Nico Williams nico at cryptonector.com
Fri May 20 18:18:16 EDT 2011


On Fri, May 20, 2011 at 4:30 PM,
<travis+ml-rbcryptography at subspacefield.org> wrote:
> Just a quick thought, I noticed the other day that rsync uses a
> "rolling MD4 hash" or something like that to detect changes in a
> window of data.

A quick look around should tell you that it uses a "rolling checksum"
and a hash function.  MD4 is one such hash function.  The rolling
checksum is a CRC, which is to say, not a hash function.

> I wonder if A/V shouldn't use something similar?

The rsync rolling CRC is useful for detecting insertions an deletions
-- i.e., remote diff.

> I assume MD4 is an outdated choice - perhaps some cryppie needs to
> design a hash function that is specifically designed for a FIFO kind
> of window?  Maybe there is and I'm just out of the loop.

MD4 isn't the function with the "rolling" property.  A function with
that property isn't a hash function.  It might be a strong CRC though,
which might be good enough for error detection, or not.

> Potentially another application is for "metadata silvering" on file
> systems like ZFS, where we want to keep an updated checksum for a
> file, to detect corruption, but still want to have, say, efficient
> writing to the file - can you support appending?  How about random access?

That would be nice, but I don't think a CRC offers strong enough
protection.  What might be nice is if the filesystem could export an
API for getting rolling CRC data -- it might speed up rsync-like
applications.  I filed an RFE for that in ZFS back at Sun (Oracle)
years ago, and I posted about it back in 2005 in this thread:

https://www.opensolaris.org/jive/thread.jspa?threadID=12917

Nico
--



More information about the cryptography mailing list