When should preserved data be allowed to die?

After a certain period of time, all data loses value. For example, the IRS only requires a few years of tax information be retained, beyond which they cannot audit by law.

At what point should "preservation" be changed into "conservation" - ie, when should data no longer be considered vital and allowed to die (or be actively deleted)?

^~related\\ to,\\ but\\ different\\ from,\\ How\\ to\\ handle\\ sensibly\\ lossful\\ digital\\ long\\ term\\ preservation?~^

warren

storage
lifetime

Comments

John Lovejoy: The issue of 'value' of the information should always be considered. There is no use preserving stuff that has no value. Records Managers and Archivists should be familiar wit the concept of "appraisal"
walker: I feel this is more discussion topic than question and answer. Without a specific context - which records to keep for how long and for whom - I don't think answering this question (which is a general appraisal question as @john-lovejoy states) will be helpful for the vast majority of visitors. Even if it were, I'm not sure appraisal questions are in scope here.

Answer by wizzard0

Since the storage capacities and data volumes have been growing exponentially, you can easily see for yourself that the cost of storing data indefinitely is limited.

I.e. you need \$1000 for storing 1 year, \$1500 for storing 2 years, \$1750 for 3 years and so on - and \$2000 for storing data indefinitely.

So, my personal opinion observerd on a number of projects is that there is no need to delete (or even support deletion in the system design) because the cost decreases together or faster than the value does.

Comments

Andy Jackson: It is hardly certain that this trend will continue indefinitely, see this analysis for example.
wizzard0: Insightful research, thanks!

Answer by Trevor Owens

The notion of Archival appraisal is relevant here. Organizations need to make decisions about the long term value of materials. Some materials have permanent value, some are set to specific retention schedules. In any case, it is a matter of an organization identifying why something is important and what uses it can serve in the future and then deciding how to manage content toward those ends.

As a side note, all data does not lose its value. There are a lot of things that weren't thought of as valuable at all but were kept around as a result of benign neglect (for example Martha Ballard's Diary). There are also things like antiquarian books, or early drafts of scientific papers, that have enormous historical value. The point here is that value is always dependent on some intended use. So the question is, how is the data useful to you and your organization and who else might it be useful to in the future.

Comments

Answer by gmcgath

There's a distinction between preservation and retention, which Trevor alludes to but should be made more explicit. Tax records are an example of retention; they're kept for a finite time for a specific need, and after that they have no value or may even be problematic to keep because of privacy issues. Records of this type normally aren't made public. Things like customer records and maintenance logs fall into the category of retention. Information which is appropriate for public disclosure falls into the "preservation" category, and in general there's no reason to delete it unless you're facing a resource crunch.