[cryptography] rolling hashes, EDC/ECC vs MAC/MIC, etc.
nico at cryptonector.com
Fri May 20 18:18:16 EDT 2011
On Fri, May 20, 2011 at 4:30 PM,
<travis+ml-rbcryptography at subspacefield.org> wrote:
> Just a quick thought, I noticed the other day that rsync uses a
> "rolling MD4 hash" or something like that to detect changes in a
> window of data.
A quick look around should tell you that it uses a "rolling checksum"
and a hash function. MD4 is one such hash function. The rolling
checksum is a CRC, which is to say, not a hash function.
> I wonder if A/V shouldn't use something similar?
The rsync rolling CRC is useful for detecting insertions an deletions
-- i.e., remote diff.
> I assume MD4 is an outdated choice - perhaps some cryppie needs to
> design a hash function that is specifically designed for a FIFO kind
> of window? Maybe there is and I'm just out of the loop.
MD4 isn't the function with the "rolling" property. A function with
that property isn't a hash function. It might be a strong CRC though,
which might be good enough for error detection, or not.
> Potentially another application is for "metadata silvering" on file
> systems like ZFS, where we want to keep an updated checksum for a
> file, to detect corruption, but still want to have, say, efficient
> writing to the file - can you support appending? How about random access?
That would be nice, but I don't think a CRC offers strong enough
protection. What might be nice is if the filesystem could export an
API for getting rolling CRC data -- it might speed up rsync-like
applications. I filed an RFE for that in ZFS back at Sun (Oracle)
years ago, and I posted about it back in 2005 in this thread:
More information about the cryptography