Fixity calculation resources

When preparing a submission information package with fixity information, which hash takes more server resources to calculate, MD5 or SHA-1?

user173

• fixity
• md5
• sha-1

Comments

Answer by Michael Kjörling

SHA-1 is a much more complex hashing algorithm than is MD5, but neither of them is computationally "cheap".

That said, when running off spinning-platter hard drives and reasonably modern hardware, neither hashing algorithm is going to be the limiting factor. The limiting factor in almost any realistic case will rather be disk read throughput during the calculations.

You should rather be concerned with the risk of changes (intentional or unintentional) in data resulting in an unchanged hash value. For that, you need a long (providing enough space to minimize the risk of collisions), good (to increase the likelihood that any change results in a different hash) hashing algorithm. MD5 is cryptographically broken, and there are serious doubts about the long-term cryptographic security of SHA-1. The combination of the two, however, may very well be viable from an archive ingestion point of view (to ensure that the data received matches what was sent).

Comments

• gmcgath: I've found that, at least when running JHOVE, calculating digest values significantly slows the operation down, so it is a limiting factor. Other software may do it more efficiently.
• Michael Kjörling: @gmcgath I don't have experience with JHOVE, but would be curious to know what portion of the performance degredation is due to hash calculation, and what portion is due to having to read the data to perform the hash calculation. The latter will be exactly the same no matter which algorithm is used, although with multiple hashing algorithms the effects can be minimized by performing multiple calculations in parallell on the same input data stream. My experience (I tried this just recently) is that on consumer-level hardware, disk I/O is vastly slower than hash calculation at least for MD5.
• gmcgath: JHOVE does the digest calculation as it reads the file, not in a separate pass. A possibility is that the way it does this plays havoc with file buffering. But we're getting a bit off topic here. Take a look at the code on SourceForge if you're sufficiently curious.