Learning from failure: The case of the disappearing Web site

Learning from failure: The case of the disappearing Web site. Francine Barone, David Zeitlyn, Viktor Mayer-Schönberger. First Monday. 4 May 2015.

This paper presents the findings of the Gone Dark Project in 2014 that looks at web sites that disappear and are no longer accessible online. A study in 2012 showed that historically significant social media content decays at a rate of 11 percent within one year, and nearly 30 percent in two years (a rate of .02 percent of resources lost per day). That is compounded by link rot, which a study of academic references showed that over 70 percent of URLs in academic journals and 50 percent found in U.S. Supreme Court opinions have broken or no longer link to the original citation information. The principal concern of the study was with sites which contain substantial or significant content. To understand the the wider landscape, the findings were categorized:

Main types of sites:

1. Scientific and Academic: Databases, research tools and repositories ranging from the natural to social sciences and humanities. Losses of this type are commonly the result of the end of funding or institutional neglect
2. Political: Personal homepages of politicians, campaign pages, political speeches and/or repositories of once-public government files.
3. Historical and Cultural: Includes collated collections of historical documents, genealogies or research portals, as well as more professionally run film, video or music archives.
4. Specialized project pages or information aggregation sites
5. Social media:

Main reasons for sites disappearing:

1. Neglect: Intentional or unintentional neglect is probably the most common reason that a site disappears
2. Technical: Technical issues are usually bundled with some form of neglect or insufficient financial resources.
3. Financial: Cost of site maintenance, staff costs, etc.
4. Natural disaster: Computer hardware is susceptible to fires, floods, rioting and neglect (just as are paper files). 
5. High-risk situations: Tumultuous political climates are a nightmare for data loss.  
6. Competition between top Web companies leads to popular services that are abandoned, shut down or absorbed into a larger platform.

 Some sites are going dark across the Web without being archived. "Restoring a single site is a challenging enough task, but when a collection of related Web sites goes dark, prevention strategies are much more difficult to specify (and quantify) as losses can potentially include an entire digital ecosystem of information". Selective archiving is best undertaken by those with firsthand knowledge of essential site. Foresight and intuition for Web preservation is not always coupled with institutional or financial stability. Solutions to the problems of sites going dark will require more awareness from all parties involved. Keeping these disappearing resources requires working with content owners to find permanent homes for at-risk data.

Digital Preservation Matters.