Zombse

The Zombie Stack Exchanges That Just Won't Die

View the Project on GitHub anjackson/zombse

Tools for identifying obfuscated files, specifically password protected and encrypted formats?

I am looking for tools capable of detecting and flagging up encrypted and password protected files in any given collection.

First there is a necessity for a tool that is capable of doing this and understanding what is out there (and its range and abilities). Second there is a requirement for usability and distribution to a probable non-expert audience.

Ross Spencer

Comments

Answer by Andy Jackson

There are two ways of answering this question. Firstly, for formats that are known to support encryption (e.g. PDF, ePub, ZIP, etc.), you might expect that characterisation tools can report the presence of encryption. This is true, but in practice many of the tools perform poorly because encryption is used in two ways: as well as full encryption (requiring a specific password to unpack a bytestream), encryption can be used to support obfuscation (where the encryption algorithm does not require a password, or only requires a known, shared, password). For example, PDF uses a default empty string password in some cases, and the JHOVE characterisation tool cannot distinguish between this and full encryption.

The second way of answering this question is as follows - are there any ways of spotting encryption in general, i.e. without understanding the format of the bytestream. The answer to this appears to be 'sometimes'. Methods such as the visualisation of bytestream entropy have been used to spot encrypted regions of files, and so this kind of information-theoretical analysis may be useful in some cases. Formally, however, encryption is statistically indistinguishable from compression, and therefore this approach will lead to a lot of false-positives in the presence of compressed data.

There appear to be very few established tools that perform these tasks.

Comments

Answer by johan

For PDF I would recommend you check out Apache Preflight, which is a PDF/A-1 validator that is part of the PDFBox library. During a recent SPRUCE hackathon we used this to identify various preservation risks in PDF (any PDF, not just PDF/A!!), see link below for a write-up by the BL's Pete Cliff:

http://www.openplanetsfoundation.org/blogs/2013-03-15-pdf-eh-another-hackathon-tale

It does distinguish between the case where you need a password to open the file, and the (less serious) print/copy restrictions.

There's also this earlier report on some first tests I did with an earlier version of Preflight back in December (some of the conclusions there are now outdated because Preflight appears to have improved quite a lot since then):

http://www.openplanetsfoundation.org/blogs/2012-12-19-identification-pdf-preservation-risks-apache-preflight-first-impression

Hope this helps!

Comments

Answer by Euan

Andy's answer above is unfortunately accurate, however this may be some hope. While this does not help you right this minute, there has been some discussion in the past of using emulated environments to do this sort of testing. Under such a model the format and interaction software of the file(s) would be submitted to an emulator which would configure itself to boot that software and open the file(s). The result of this process would be analysed to see if it included a password input or decryption requirement. The whole process could be automated and set to iterate across a large set of files without manual input. This process would require very similar technology to that included in the Migration by Emulation Service developed as part of the Open Planets Foundation work.

Such a tool does assume a certain degree of knowledge of the software needed to interact with the files and this may limit its applicability. On the other hand it might be possible to pass an unknown file to a varied set of emulated environments in order to help both identify the right software and identify whether the content is password protected or encrypted.

Comments