When is it legal/ethical to preserve website content?

There is a known problem with the Wayback Machine regarding domain name ownership - oftentimes, the domain changes hands, and the new owner either puts a spam blog on it, or puts a robots.txt which forbids crawling, and then WM promptly deletes the entire website history.

Is it legal to crawl and store personal copies of websites to protect against this particular problem? or to build such an engine?

wizzard0

• web-archiving
• legal
• digital-born
• crawling
• copyright

Comments

• Nick Krabbenhoeft: I think this is better as two questions. 1. How do spam websites affect web crawlers like Heritrix? 2. What are the ethics of preserving public websites without asking permission from the site owners?
• wizzard0: I split them, thanks!
• luser droog: Can you make them look more different? It's been flagged as a duplicate.
• Andy Jackson: Can I suggest an edit under a title along the lines of 'who is legally allowed to archive website?' Or similar?
• Paul Wheatley: I agree with Andy. "Preserving website content" is not a question. needs more specific explanation in the title.
• wizzard0: Renamed the question

Answer by Andy Jackson

Whether you as a individual have the right to take a copy of a website, and for what purposes, and whether you can redistribute it, will be down to you local copyright laws. I expect fair use/fail dealing exceptions mean you can copy but not distribute, but YMMV.

Another option is to be aware that there is more than just one web archive, each with different policies, so you could check for other holdings or nominate the site for archival elsewhere.

EDIT

I notice you included the ethics issue when you updated the title. This a very good point, and one that's not really been explored in detail as far as I know (although this work on Search Engines and Ethics might be a good starting point). Certainly, personally speaking, I'm not sure large national web archives should be archiving Facebook as a matter of course (for example), any more than we should be archiving everyone's email, or digitising everybody's letters. Such personal information should be 'opt in' only, IMO.

Comments

Answer by mopennock

The ethical issues around web archiving are interesting and the ethics of web archiving can vary from case to case. To judge whether a particular case is ethical, you might want to consider:

• Why do I need to take a copy?
• Do I have or need the owner's permission?
• Do I have or should I ask for permission of anyone else who has contributed to the site?
• Is there any reason why the owner would not want me to have a copy?
• What is the subject matter?
• Who will have access to my copy?
• How long do I intend to keep it for?
• Can I keep it securely and if not, who might be affected should it be released back on to the live web?

There have been a few papers published in recent years:

• 'Ethical issues in web archive creation and usage' (2008)
• 'Personal Internet Archives and Ethics' (2012).

The latter is particularly relevant given the context stated in your question.

Comments

• Avery Payne: There are some sites that simply don't exist anymore. They were publicly run, publicly available, and had various amounts of content on them. Contacting them in this context and attempting to obtain permission to capture that information for personal purposes would be a difficult, if not impossible, process. A general rule of thumb I have used: if Archive.org won't touch it, then don't bother.