The Seven Deadly Sins of Records Retention

The Seven Deadly Sins of Records Retention.  Sarah D. Scalet. CSO Security and Risk.  July 01, 2006.

 Legal actions in the past few years have made document retention programs important. One wrong step can cost an organization money.  Some have concluded that they should archive, forever, anything and everything to be on the safe side. But keeping too much information is a risk too, as you can expose yourself to litigation risks, and possibly violating privacy rights. 

1.       Not keeping your records straight from your backup.

• The first step to a good records management program is simply identifying what a record is. E-mail servers and network drives get backed up to keep the business running. But a record is "something that you need to keep around for a set period of time, either for regulatory, legal or business reasons. Records encompass both structured information, like financial transactions ... and unstructured information, like financial spreadsheets." 
• "while backup media may be in a continual state of being written and overwritten, records that must legally be retained often need to be stored on immutable, nonrewritable storage, and should be either very well-organized, very easily searched or both.

2.       Expecting the legal department to produce a rule of thumb for how long to store records.

3.       Assuming that document retention is someone else's job.

4.       Not being able to respond quickly to a request.

5.       Having a policy you can't follow.

6.       Failing to offer guidance on how to destroy old records.

7.       Telling people to delete information at the wrong time.

Start with an accurate survey of the information that's in the organization, a data map.
"At the end of the day, you have to have some sort of written policy around it."

Digital Preservation Matters.

Online Sharing of Digital Preservation Problems and Solutions

Online Sharing of Digital Preservation Problems and Solutions. bram van der werf. Open Planets Foundation. 27 October 2011.
The vision of the Open Planets Foundation from the beginning is that meaningful discussions around actual Digital Preservation problems are key to finding solutions, tools and ultimately improving digital preservation practice.  It encourages people to document issues and search for practical solutions to the problems. 

The online digital preservation community needs to be driven by content issues, practical solutions and above all meaningful discussions between all stakeholders. Digital Preservation might be a technology challenge but it is definitely not for the sake technology, it is about keeping digital content safe and accessible. The technology serves the digital content users and curators. The digital preservation practitioners for small to medium institutions where people have multiple roles are clearly the collection owner. For larger institutions, this becomes more of a distributed role among those with different technical responsibilities. By involving all the groups everyone can benefit from the larger organizations and  can often take actions on today’s problems as experienced by practitioners of medium/small institutes.

Digital Preservation Matters.

Metadata Clean Sweep: A Digital Library Audit Project

Metadata Clean Sweep: A Digital Library Audit Project. R. Niccole Westbrook, et al.
As digital library collections grow in size, metadata issues such as inconsistencies, incompleteness and quality become increasingly difficult to manage over time.  This was a project by library school interns and full-time staff to alleviate poor recall, poor precision and metadata inconsistencies across digital collections. Access to content is the primary goal of the metadata of a digital library. Successful metadata maintenance must include both correction of found mistakes and correction to practices that lead to those mistakes.

Digital Preservation Matters.

A simple JP2 file structure checker.

A simple JP2 file structure checker. Johan van der Knijff. Open Planets. 1 September 2011.
One major risk of file migration is that hardware failures during the migration process may result in corrupted images. Ideally JHOVE would be able to detect such errors using format validation tools.  "I started out with removing some trailing bytes from a lossily compressed JP2 image. As it turned out, I could remove most of the image code stream (reducing the original 2 MB image to a mere 4 kilobytes!), but JHOVE would still say the file was "well-formed and valid". I was also able to open and render these files with viewer applications such as Adobe Photoshop, Kakadu's viewer and Irfanview. The behaviour of the viewer apps isn't really a surprise, since the ability to render an image without having to load the entire code stream is actually one of the features that make JPEG 2000 so interesting for many access applications. JHOVE's behaviour was a bit more surprising, and perhaps slightly worrying." So he wrote a very simple JP2 file structure checker that is available for download.

Digital Preservation Matters.

 

The CLIF Project: The Repository as Part of a Content Lifecycle

The CLIF Project: The Repository as Part of a Content Lifecycle. Richard Green, Chris Awre, Simon Waddington. Ariadne. 9 March 2012.
This was a joint project that did an extensive literature review and worked with  digital content creators to understand how to deal with the interaction of the authoring, collaboration and delivery of materials. At the heart of meeting institutional requirements for managing digital content is the need to understand the different operations through which content goes, from planning and creation through to disposal or preservation. Repositories must be integrated with the other systems that support other parts of this lifecycle to prevent them becoming yet another information silo within the institution.

The CLIF software has been designed to try and allow the maximum flexibility in how and when users can transfer material from one system to another, integrating the tools in such a way that they seem to be natural extensions of the basic systems.  This open source software is available for others to investigate and use.

The repository’s archival capability is regarded as one of its strongest assets, and the role of the repository within a University will be regarded very much in terms of what it can offer that other campus systems cannot.  It should not try to compete on all levels. There is a need to clarify better at an institutional level what functionality is offered by different content management systems, in order to better understand how different stages of the digital content lifecycle can be best enabled.

Digital Preservation Matters.

Pogoplug service turns your computers into private cloud

Pogoplug service turns your computers into private cloud. Lucas Mearian. Computerworld. May 9, 2012.
Pogoplug Team which allows home offices and small businesses (best for 10 - 50 people) to turn file servers or spare PCs into pools of storage accessible from the Web. This basically creates a private cloud for businesses so they can store information within their own firewalls rather than on third-party services. The product allows multiple users to share storage in a home or office. The service's capabilities are similar to Dropbox for file sharing and collaboration. It has full access and file sharing to 3TB of storage for $150 (not including the $15-a-year, per-person licensing fee paid to Pogoplug for its service.)

A prototype JP2 validator and properties extractor.

A prototype JP2 validator and properties extractor. Johan van der Knijff. The Open Planets Foundation.  14 December 2011, Update March 2012.   

The site includes a discussion of the full-fledged JP2 validator tool, as well as how it validates the object. A prototype is now ready in the form of the jpylyzer tool. The tool is both a validator and a properties extractor.

Digital Preservation Matters.

Bit Rot & Long Term Access.

Bit Rot & Long Term Access. bram van der werf. Open Planets. 28 February 2011.
Discussion of bit rot in an image collection and the call to examine more closely preservation and long term access. There is a Problem, Requirement, Use Case sequence in the "what we need" section of the OPF wiki.
"We need to make serious efforts to drill down into every individual potential element and risk for Bit Rot. Create Use Cases and take action on these Use Cases and develop practical tools and/or workflow to assure we can prevent Bit Rot." These may be more a case of creating scripts that can compare "checksums" or "open and save as new" type of activities inside a repository.  Computer hardware is "subject to a very short technical life cycle and by far not as reliable for keeping the integrity of the bits as we sometimes like to believe."
It is crucial to understand the challenge and for experts to open a continuous discussion.

BitCurator: Tools and Techniques for Digital Forensics in Collecting Institutions.

BitCurator: Tools and Techniques for Digital Forensics inCollecting Institutions.  Christopher A. Lee, et al.  D-Lib Magazine.  May/June 2012.

The BitCurator Project aims to incorporate digital forensics tools and methods into collecting institutions' workflows for libraries, archives and museums (LAMs).  There is an increasing need to move born-digital materials from removable media into more sustainable preservation environments.  The project criteria for the multi-year project will include the following:

• The forensic technologies should be open source, extensible, and mature.
• Development will focus on extensions, plugins, and wrappers for proven software rather than from-scratch development.
• It will adhere to a common digital forensic metadata standard and provide crosswalks to relevant library and archives metadata schemas.

It will work with Advanced Forensic Format and Digital Forensics XML (DFXML).   

Digital Preservation Matters.

Implementing DOIs for Research Data.

Implementing DOIs for Research Data.  Natasha Simons. D-Lib Magazine. May/June 2012.
As research becomes more collaborative and global it is also becoming more difficult to manage the large amounts of research data generated daily.  The Digital Object Identifier (DOI) system is one way to create persistent identifiers for research data collections and datasets. "Data that is richly described, organised, integrated and connected allows the data to be more easily discovered by other researchers." Identifying such resources allow research data collections and datasets be open and discoverable to others, but there are questions that need to be answered, such as the type of material to get a persistent id, the granularity, whether the landing page or the resource should get the id, who creates and maintains the ids, and for how long. The questions, common to other institutions, should encourage discussion and collaboration.

Digital Preservation Matters.

Preserving Email DPC Technology Watch Report released.

Preserving Email DPC Technology Watch Report released. Neil Beagrie.  Blog. News release. 17 Feb 2012.  
The 57 page Preserving Email technology watch report gives practical advice on how to ensure that  email remains accessible.   "Users normally shoulder the ultimate responsibility for managing and preserving their own email.  This exposes important records to needless risks and is counterproductive in many cases." The three basic steps which institutions should undertake:

1. Define policies, including institutional commitment, specific actions to take, and end-user expectations, responsibilities and rights regarding the email archives
2. choosing appropriate tools to work with email in an environment where users have adequate storage space and without auto-deletion settings
3. implementing them in the light of local environmental factors and available resources.

 The report suggests five different preservation strategies with possible tools (such as Aid4Mail and Emailchemy) or methods (co-managing archive folders) in some of the cases, for the institution and individuals.

1. The ‘sweeping up crumbs’ or whole-account approach refers to harvesting email found on a user’s computer or account. 
2. The ‘nurturing and harvesting’ approach, helping email users ensure that critical records are retained in system-neutral formats, then using email migration software to capture and preserve records either as they are created or at the end of a user’s lifetime. An example is providing users a designated ‘archives’ box.
3. The ‘capturing carbon’ or whole system approach implements email archiving software to capture an entire email ecosystem or a portion of that ecosystem to an external email storage environment.
4. The ‘tagging and bagging’ or message-by-message approach, existing electronic record management systems, but which may not be effective.
5. The Personal Archives Service approach, which would offer services such as Carbonite or CrashPlan.

Unless we make the preservation of trusted email records a systematic part of our everyday operations, many important records will be lost. They cite some examples, such as the 22 million emails from the Executive Office of the President of the United States surrounding the Gulf War that we lost when servers were replaced. 

Digital Preservation Matters.

Lessons From a (Former?) Digital Preservation Neophyte.

Lessons From a (Former?) Digital Preservation Neophyte. Chris Prom. Website. April 10, 2012.

What do we really need to do:

1. Render objects
2.  Retain evidence
3. Show authenticity

We do not need to understand it all at once, nor implement it all at the same time

1. Get your own house in order
2.  Begin helping others
3. Develop a digital program statement
4. Acquire and deal with digital born stuff
5.  Begin to implement a TDR

General Process:

• Understand your limitations
• Understand the format and structure of what you are trying to preserve
•  Match tools to desired services
• Be consistent
•  Be willing to experiment (and fail)
•  Provide something other people need

• The process of learning an art can be divided conveniently into two parts: one, the mastery of the theory the other, the mastery of the practice. Erich Fromm
• Thought can only lead us to the knowledge but  it cannot give us the ultimate answer.  The only way in which the world can ultimately be grasped lies not in thought, but in the act.”

 

Digital Preservation Matters.

The Digital Preservation Network.

The Digital Preservation Network. George Eberhart. American Libraries Magazine. May 4, 2012.
The ARL membership meeting discussed the recently established Digital Preservation Network, a federation of universities intent on securing the long-term preservation of the digital scholarly record.  James Hilton said that only universities, not private industry, can solve the problem of preserving born-digital data and making it accessible to future generations. “Universities last for centuries. Companies do not.”
Other notes from the report:

• The problem is too large for just libraries to handle. The presidents of universities must be on board with this.
• The data and metadata of research and scholarship are susceptible to multiple single points of technical, political, and funding failure.
• The program asks for a $20,000 initial commitment to help fund project software.

Digital Preservation Matters.

Digital Preservation Tools.

Digital Preservation Tools.  Paul Wheatley. SPRUCE Project Wiki. May 10, 2012.
Open Planets Foundation Digital Preservation Tool Registry. This lists many tools of all types that may be just what you are looking for, and also references to additional tool lists, such as http://agogified.com/tools-and-services.  A few of the categories include:

• Transcription Tools
• DataOne
• Forensic tools
• PDF Creation
• File Conversion tools
• Audio file conversion tools
• Web capture
• Email mining tools
• Record analysis tools
• Curation and data management tools
• Bulk extractor tools
• etc.

Digital Preservation Matters. 

Digital Preservation: Question and Response

Question: We are trying to move away from multiple physical copies of our archival audio.  Our goal is to just have redundant digital copies and one physical copy on some medium in case of catastrophic failure with the servers. What physical medium other audio archives are using for preservation/backup copies of digital audio files.

Response: It is important to have redundant digital copies in case of failure; we have had a number of occasions where we needed to recover objects from a second or third copy when other copies failed.  There are different ways to accomplish multiple digital copies, such as copies on server, tape (LTO), cloud (DuraCloud and others), and optical discs, or a mixture. There are advantages and disadvantages with each option; we have a mixture of these, but are moving towards two options, server copies with tape backup, and M-DISCs.  We have started using the Rosetta software to manage the server copies, which will have tape copies stored in a secure off site location.  But I want an additional copy on another medium, which is our M-DISC archive.

In general optical CDs and DVDs have a short lifespan.  For a number of years, I have been caring for a large disc archive of gold CDs and DVDs; there are multiple copies for redundancy. (There are tape copies also, which have had problems as well, but the gold archival discs have been considered the preservation copies.) Multiple copies are necessary since a percentage of the discs fail each year (the percentage depends on the collection).  In order to solve that problem, two professors on campus (Information Technology, and Chemistry & Material Science) created a digital storage medium that does not fail over time and is unaffected by any normal factors, such as light, heat, cold, oxidation, pollutants, material decay, bit flips, etc.  (Extreme stress tests by the US military could not get the discs to fail.)

The university licensed the technology to a company called Millenniata which has partnered with LG and others to produce the M-DISC.  Currently the M-DISC is a DVD format, which is somewhat of a drawback if you want to store large archives, but they are developing other densities and options. (There are some organizations using a multi-terabyte Millenniata device, but I have not seen it.) I have tested and used the M-DISCs for several years and have not had any problems.  (I plan to start another round of testing on my M-DISC archive this year to check on the status of the discs; I check them for usability, read error levels, and bit transfer integrity.) So I consider this my ‘copy in case of catastrophic failure’, while the Rosetta archive is my ‘active preservation archive’. There are others in the library and on campus (digital lab, records management, etc) that use the M-DISCs for their own long term copy for images, documents, audio, video, and such. Whatever you choose, you should think about multiple copies in multiple places on multiple media.  What you choose needs to fit your circumstances and be sustainable for your program. 

 Digital Preservation Matters.

Applying to NEH's Preservation Assistance Grants - Webinar.

Applying to NEH's Preservation Assistance Grants - Webinar. Elizabeth Joffrion. National Endowment for the Humanities.March 12, 2012.
An excellent webinar for the Connecting to Collections community on the NEH's Preservation Assistance Grant, what they are, how to apply for them, and examples of some projects that have been funded, including BYU's grant for a digital preservation workshop. Includes an introduction to the grants; who qualifies as a smaller institution; proposal development and grant writing strategies.They encourage institutions that have never applied before. The PAG grants are easy to write and there is no sharing cost.

 Digital Preservation Matters.

Library of Congress Digital Preservation Newsletter.

Library of Congress Digital Preservation Newsletter. May 2012. [PDF]
Items from the Newsletter include:

• Key outcomes of the NDIIPP program are to identify priorities for born digital collections and engage organizations committed to preserving digital content.
• Viewshare  is being used for the collections
• Floppy Disks are Dead, Long Live Floppy Disks
• Floppy disks are fragile constructions that were never designed for permanence.
• Difficult to determine what is on the floppy and to recover
• A floppy disk controller called Catweasel allows computers to access a wide variety of older disk formats (must have the floppy drive).
• Web archiving.  
• Because of the scope of the web sites, consider partnering with other institutions.
•  Preservation of and Access to Federally Funded Scientific Data
• Research data produced by federally funded scientific projects should be freely available to the wider research community and the public
• Public data should be a public resource, and data sharing supports core scientific values like openness, transparency, and replication. 
• Lack of resources for curating scientific data and a lingering tradition of data hoarding create resistance to open access to research data.

 Digital Preservation Matters.

Guidelines on the Production and Preservation of Digital Audio.

Guidelines on the Production and Preservation of Digital Audio. The International Association of Sound and Audiovisual Archives. ed. by Kevin Bradley. Second edition 2009.
A web version of guidelines on creating and preserving digital audio objects.  Areas of interest covered include:

• Metadata, including preservation metadata
• Important to keep all aspects of preservation and transfer relating to audio files, including all technical parameters 
• Persistent Identifiers
• Naming conventions
• Digital preservation planning
• OAIS: the services and functions for monitoring the environment and providing recommendations to ensure that the information stored remains accessible to the Designated User Community over the long term, even if the original computing environment becomes obsolete
• Preservation planning is the process of knowing the technical issues in the repository, identifying the future preservation direction (pathways), and determining when a preservation action, such as format migration, will need to be made.
• Small scale digital archiving
• Digital preservation is as much an economic issue as a technical one.
• Any proposal to build and manage an archive of digital audio objects should have a strategy which includes plans for the funding of ongoing maintenance and replacement
• Risks
• The only way to know the condition of a digital collection is constant and comprehensive testing.
• A preservation strategy requires a listing of the risks associated with the loss of technical expertise and how that will be addressed.   
• An archive can distribute risk in a number of ways,  such as:
• form local partnerships so that content is distributed
• establish a relationship with a stable well funded archive; 
• engage a commercial supplier of storage services
• Recording formats
• It is not recommended that audio streams be recorded for long term storage.
• Recommend recording as a digital data file
• Recommend recording as .wav or preferably BWF .wav files 
• The large investment in a single format will also help support the continuance of that format for the longest period, as the industry will not change an entrenched format without significant benefits.
  
  Digital Preservation Matters.

New DuraCloud Digital Archiving and Preservation Services.

New DuraCloud Digital Archiving and Preservation Services. Press Release. Duraspace. May 1, 2012.
The DuraSpace organization announced new DuraCloud subscription plans that offer three levels digital preservation and archiving services in the cloud. Prices for the new subscription plans are competitive with commercial cloud providers and do not require additional transfer or variable costs.

1. DuraCloud Preservation Basic: for institutions that would like to have two redundant copies of their original content stored at one cloud data center. 
2. DuraCloud Preservation Plus: for institutions that wish to have four redundant copies of their original content stored at two cloud data centers. 
3. DuraCloud Enterprise: a full suite of configurable DuraCloud features for institutions that need multiple DuraCloud sub-accounts for departments, research groups, cross institutional projects, or individuals.

DuraCloud offers automated weekly content health checks, reporting, file repair  and more. It  runs on Amazon AWS, Rackspace and in the near future on the San Diego Supercomputer Center (SDSC) Cloud, and provides customers with:
• Copies of the content stored with multiple providers
• Automated health checking of content
• Automated repair of damaged files for Preservation Plus customers
• A full suite of reports
• Online sharing and streaming to any internet-linked device

Digital Preservation Matters.

Preserving Moving Pictures and Sound.

Preserving Moving Pictures and Sound. Richard Wright. DPC Technology Watch Report 12-01. March 2012. [PDF]
This excellent report is for anyone with responsibility for collections of sound or moving image content and an interest in preservation of that content. For audiovisual materials, digitization is critical to the survival of the content because of the obsolescence of playback equipment and decay and damage of physical items, whether analogue or digital.

The basic technology issue for audio/visual content is to digitize all items on the shelves, either for preservation or access. The risk of loss is high. Another issue is moving content from the current media to digital files. A third issue is preserving the digital files. This report describes the techniques for preservation planning, digitization and digital preservation of audiovisual content, and describes the technologies.  Preservation of these materials is difficult because they are physically, culturally, and economically different.
Explanation of signals and carriers. "Digital technology produces recordings that are independent of carriers. Carrier independence is liberation". Digital preservation of the digitized signal means to preserve the numbers, but also the technology to decode the numbers. ‘Maximum integrity’ means keeping the full quality of the audio and video. As far as possible, the new preservation copy should be an exact replica of the original: the content should not be modified in any way’. This may be difficult to achieve.

The two basic kinds of preservation action are: 1) changing the audiovisual content within a collection, or normalization; 2) changing the system that holds the collection.
There are four main factors in an analogue or digital conservation program:

1. packaging (wrappers), handling and storing;
2.  environmental conditions;
3.  protecting the masters; and
4.  condition monitoring, maintaining quality.

The four PrestoPRIME requirements for effective access to time-based media are:

1. granularity: division of the content into meaningful units;
2. navigation: the ability to select and use just one unit,
3. citation: the ability to cite a point on the time dimension of an audio or video file, with a permanent link
4. annotation: the ability of a user of content to make time-based contributions

Other topics include access rights, implications for small collections or institutions. The digitization standards, encoding, wrapper and metadata are all agreed and well documented.  There is no reason for the basic encoding to ever be changed, though wrappers may eventually become obsolete. All archives need to be aware of the risk of loss of embedded metadata. Finally, surveys have shown that in universities there is a major problem of material that is scattered, unidentified, undocumented and not under any form of preservation plan.

Digital Preservation Matters.

University of Utah Selects Ex Libris Rosetta for Long-Term Digital Preservation

University of Utah Selects Ex Libris Rosetta for Long-Term Digital Preservation. Press Release. April 30, 2012.
Ex Libris is pleased to announce that the University of Utah has opted for Ex Libris Rosetta to preserve the school's extensive cultural heritage collections, which include newspapers and other historical textual documents, photographs, rare books, oral history interviews (including transcripts and audio), motion picture collections, and more. In addition to cultural heritage collections, Rosetta will enable the University of Utah to preserve faculty publications and research data. The J. Willard Marriott Library hosts the collections of many campus departments and, as a member of the Mountain West Digital Library network, hosts collections belonging to other Utah institutions.

Digital Preservation Matters.

An Overview of Web Archiving.

An Overview of Web Archiving. Jinfang Niu. D-Lib Magazine. March/April 2012.
An article on the methods used at a variety of universities, and other institutions to select, acquire, describe and access web resources for their archives. Some notes from the article:

• Web archiving is the process of gathering up data that has been recorded on the World Wide Web, storing it, ensuring the data is preserved in an archive, and making the collected data available for future research.
• The workflow of web archiving includes appraisal and selection, acquisition, organization and storage, description and access. This workflow is the core of web archiving.
• Creating a web archive presents many challenges,  
• When archiving web content through web crawling programs, selection decisions are the basis for compiling a site list to crawl and configuring crawler parameters. Crawling may replace deposit for some things.
• In acquiring web resources, the decision of whether to seek permission from copyright owners depends on the legal environment of the web archive, the scale of the web archive, and the nature of archived content and the archiving organization.
•  Web archives need to preserve the authenticity and integrity of archived web content. The concept of provenance is important. 
• The library must decide how it will generate, store and use metadata. Also, how it will make this available to others.

Digital Preservation Matters.

Web Archives for Researchers: Representations, Expectations and Potential Uses.

Web Archives for Researchers: Representations, Expectations and Potential Uses. Peter Stirling, et al. D-Lib Magazine. March/April 2012.
Web archiving is one of the missions of the Bibliothèque nationale de France. This study looks at content and selection policy, services and promotion, and the role of communities and cooperation.  While the interest of maintaining the "memory" of the web is obvious to the researchers, they are faced with the difficulty of defining, in what is a seemingly limitless space, meaningful collections of documents. Cultural heritage institutions such as national libraries are perceived as trusted third parties capable of creating rationally-constructed and well-documented collections, but such archives raise certain ethical and methodological questions.

To find source material on the web, some researchers look for non-traditional sources, such as blogs and social networks.  Researchers recognize the value of web archives, especially because websites disappear or change quickly.  The Internet is no longer just a place for publishing things, “but rather the traces left by actions that people could equally perform in the streets or in a shop: talking to people, walking, buying things... It can seem improper to some to archive anything relating to this kind of individual activity. On the other hand, one of the researchers acknowledges that archiving this material would provide a rich source for research in the future, and thus compares archiving it to archaeology.”  Some ask, "How do you archive the flow of time?" New models may be needed. And when selecting an archive, the selection criteria should also be archived, as they may change over time.

Digital Preservation Matters.

The secrets of Digitalkoot: Lessons learned crowdsourcing data entry to 50,000 people (for free).

The secrets of Digitalkoot: Lessons learned crowdsourcing data entry to 50,000 people (for free). Tommaso De Benetti.  Microtask. June 16, 2011.
National Library of Finland launched a project called Digitalkoot, which was a test of crowdsourcing with 50,000 volunteers.  The aim was to digitize the National Library’s archives and make them searchable over the internet. The volunteers input data that Optical Character Recognition (OCR) software struggles with (for example documents that are handwritten or printed in old fonts). Digitalkoot relies on machines, humans and a gaming twist.

Digital Preservation Matters.

Dream of perpetual access comes true!

Dream of perpetual access comes true! Jeffrey van der Hoeven. Open Planets Foundation.  29 March 2012.
The KEEP project released its final version of the open source Emulation Framework software. This project has brought emulation in the digital preservation context to the next level, that is, user friendly.  The easy to install package runs on all major computer platforms.  It automates several steps:

1. identify what kind of digital file you want to render;
2. find the required software and computer platform you need;
3. match the requirements with available software and emulators;
4. install the emulator;
5. configure the emulator and prepare software environment;
6. inject the digital file you selected into the emulated environment;
7. give you control over the emulated environment.

 The software supports six different computer platforms out of the box: x86, Commodore 64, Amiga, BBC Micro, Amstrad, Thomson, by using seven open source emulators which are distributed  with the Emulation Framework. 

Digital Preservation Matters.

With tech breakthrough, Seagate promises 60TB drives this decade

With tech breakthrough, Seagate promises 60TB drives this decade.  Lucas Mearian. Computerworld.  March 20, 2012.

 Seagate said they have achieved a density of 1 terabit (1 trillion bits) per square inch on a disk drive platter. The technology would lead to the production this decade of 3.5-in. hard drives with up to 60TB of capacity.

As drive manufacturers store more bits per square inch on the surface of a disk, they also tighten the data tracks. The challenge as those tracks tighten is overcoming magnetic disruption between the bits of data, which causes bits to flip their magnetic poles resulting in data errors.

Digital Preservation Matters.

Brigham Young University Selects Ex Libris Rosetta

Brigham Young University Selects Ex Libris Rosetta. Press Release. February 14, 2012.
Ex Libris has announced that Brigham Young University (BYU) has selected the Ex Libris Rosetta digital preservation system. The first member of the Association of Research Libraries (ARL) to adopt Rosetta, Brigham Young will implement Rosetta initially for the Harold B. Lee Library’s digital collections and will later expand the implementation to the digital assets of the university’s colleges and schools.

Rosetta encompasses the entire digital preservation workflow, including the acquisition, validation, ingest, storage, preservation, and delivery of digital objects. Rosetta enables academic institutions as well as libraries, archives, and other memory institutions to manage, preserve, and provide access to institutional documents, research output in digital formats, digital images, Web sites, and other digitally born and digitized materials.

Digital Preservation Matters. 

Why don't we already have an Integrated Framework for the Publication and Preservation of all Data Products?

Why don't we already have an Integrated Framework for the Publication and Preservation of all Data Products?   Alberto Accomazzi,et al. Astronomical Data Analysis Software and Systems.  
7 Dec 2011.
Astronomy has long had a working network of archives supporting the curation of publications and data. There are examples of websites giving access to data sets, but they are sometimes short lived.  "We can only realistically take implicit promises of long-term data archival as what they are: well-intentioned plans which are contingent on a number of factors, some of which are out of our control." We should take steps to ensure that our system of archiving, sharing and linking resources is as resilient as it can be.  Some ideas are: 

1. future-proof the naming system: assign persistent data IDs to items we want to preserve 
2. provide the ability to cite complete datasets, just as we can cite websites
3. include a data reference section in academic papers

Curated datasets need to be preserved indefinitely for scholarly purposes.

Digital Preservation Matters.

A literature review: What exactly should we preserve? How scholars address this question and where is the gap

A literature review: What exactly should we preserve? How scholars address this question and where is the gap.  Jyue Tyan Low. Cornell University Library. 7 Dec 2011.
There are generally two approaches to long-term preservation of digital materials

1. preserving the object in its original form as much as possible along with the accompanying systems,
2. migration or transformation: transforming the object to make it compatible with more current systems but retaining the original “look and feel.

Migration is the most widely used method, but there can be changes to the original.  If some of the original properties are lost, what then are the essential properties to maintaining its integrity?  Currently there are no formal and objective way to help stakeholders decide what the significant properties of the objects are, which are defined as:

The characteristics of digital objects that must be preserved over time in
order to ensure the continued accessibility, usability, and meaning of the
objects, and their capacity to be accepted as evidence of what they purport
to record.

An important goal of digital preservation is more than just retrieving the objects, it is to ensure the authenticity of the information.  A digital object can change as long as the final output is what it is expected to be.  The properties to preserve come from the purpose of the object, and at least one purpose for the object needs to be defined. Archivists have created standards that look at records in the context of their creation, intended use and preservation.  It is important to ask what features of the object is important when delivering to the user.  There may be many uses to many communities that were not intended by the object creator, so we should not let the ideal limit the reasonable.

Digital Preservation Matters.

Geospatial Data Preservation.

Geospatial Data Preservation. Website. November 2011.
The Geospatial Resource Center is being developed as a finding tool for freely available web-based resources about the preservation of geospatial information. A variety of selected resources are being added, including reports, presentations, standards, and information about tools for preparing geospatial assets for long-term access and use. The resources are indexed to enable searching of titles and are categorized to facilitate discovery by choosing among topics, resource types, or both. The website contains many valuable resources.  A few resources from these three categories:

Education & Training

• Appraisal and selection of geospatial data for preservation
• Best Practices for Geospatial Programs
• Copyright Quickguide

Tools & Software

• Cost Estimation Toolkit
• Conversion tools for geospatial data
• Geospatial metadata tools

Policies & Benefits

• Collection policies,
• Content standards,
• Policies on Open geospatial data access and preservation

OAIS / TDR presentation at FDLP.

OAIS / TDR presentation at FDLP.  James A. Jacobs. Federal Depository Library Conference. Free Government Information. October 2011. [PDF]
A presentation giving an introduction to the "Reference Model for an Open Archival Information System" (OAIS) and the "Audit And Certification Of Trustworthy Digital Repositories" (TDR).  This includes slides with speaker notes and a nice handout about related information with links. Every library decision should assess the impact of digital issues.  Notes:

OAIS
1. It defines the functional concepts of a long-term archive with consistent, unambiguous terminology.
2. It gives us a functional framework for designing archives, and a functional model.
3. It gives us a standard for “conformance.”
4. It is a “Reference Model” that describes functions; it is not an “implementation”
5. Some key OAIS concepts are:
   - Designated Community: An identified group of potential Consumers who should
      be able to understand a particular set of information.
   - Description of roles and functions in the information life cycle.
   - The Long Term: Long enough for there to be concern about changing technologies,
      new media and data formats, and a changing user community.
   - Preserved content must be usable according to the designated community

TDR
Documents what is being done and how well it is being done.
Provides 109 “metrics” for measuring conformance to OAIS in three areas: 
1. Organizational Infrastructure
2. Sustainability and succession plan
3. Digital Object Management
4. Technical Infrastructure And Security Risk Management

Cite Datasets and Link to Publications

Cite Datasets and Link to Publications. Digital Curation Centre. 18 October 2011.
The DCC has published a guide to help authors / researchers create links between their academic publications and the underlying datasets.  It is important for those reading the publication to be able to locate the dataset.  This recognizes that data generated during research are just as valuable to the ongoing academic discourse as papers and monographs, and in many cases the data needs to be shared. "Ultimately, bibliographic links between datasets and papers are a necessary step if the culture of the scientific and research community as a whole is to shift towards data sharing, increasing the rapidity and transparency with which science advances."

This guide has identified a set of requirements for dataset citations and any services set up to support them. Citations must be able to uniquely identify the object cited, identify the whole dataset and subsets as well.  The citation must be able to be used by people and software tools alike.  There are a number of elements needed, but the "most important of these elements – the ones that should be present in any citation – are the author, the title and date, and the location. These give due credit, allow the reader to judge the relevance of the data, and permit access the data, respectively."  A persistent url is needed, and there are several types that can be used. 

Audit And Certification Of Trustworthy Digital Repositories.

Audit And Certification Of Trustworthy Digital Repositories. CCSDS. September 2011. [PDF, 77pp.]

The Management Council of the Consultative Committee for Space Data Systems (CCSDS) has published this manual of recommended practices. It is based on the 2003 version from RLG. “The purpose of this document is to define a CCSDS Recommended Practice on which to base an audit and certification process for assessing the trustworthiness of digital repositories. The scope of application of this document is the entire range of digital repositories.”

The document addresses audit and certification criteria, organizational infrastructure, digital object management, and risk management.  It is a standard for those who audit repositories; and, for those who are responsible for the repositories, it is an objective tool they can use to evaluate the trustworthiness of the repository.

National Archives Digitization Tools Now on GitHub

National Archives Digitization Tools Now on GitHub. NARAtions. October 18, 2011.
The National Archives has begun to share applications developed in-house to facilitate digitization workflows. These applications have significantly increased the productivity and improved the accuracy and completeness of the digitization.Two digitization applications, “File Analyzer and Metadata Harvester” and “Video Frame Analyzer” are publicly available on GitHub.

• File Analyzer and Metadata Harvester: This allows a user to analyze the contents of a file system or external drive and generate statistics about the contents, generate checksums, and verify that there is a one-to-one match of before and after files. The File Analyzer can import data in a spreadsheet, and can match and merge results with auxiliary data from an external spreadsheet or finding aid.
• Video Frame Analyzer: This is used to objectively analyze technical properties of individual frames of a video file in order to detect quality issues within digitized video files.  It reduced the time to do quality checks by 50%. 

Research Librarians Consider the Risks and Rewards of Collaboration.

Research Librarians Consider the Risks and Rewards of Collaboration. Jennifer Howard. The Chronicle of Higher Education. October 16, 2011.

Association of Research Libraries’ meeting discussed research and preservation projects like the HathiTrust digital repository and the proposed Digital Public Library of America, plans for which are moving ahead. Concerning the Digital Public Library of America: “Library” is a misnomer in this case, which is more of a federation of existing objects. It wouldn’t own anything. The main contribution would be to set standards and link resources.  “The user has to drive this.”

They said that it’s almost three times more expensive to store materials locally than it is to store them with HathiTrust. Researchers now also create and share digital resources themselves via social-publishing sites such as Scribd. There is a need for collection-level tools that allow scholars and curators to see beyond catalog records.

Discussed Recollection, a free platform built by NDIIPP and a company named Zepheira to give a better “collection-level view” of libraries’ holdings. The platform can be used to build interactive maps, timelines, and other interfaces from descriptive metadata and other information in library catalogs. So, for instance, plain-text place names on a spreadsheet can be turned into points of latitude and longitude and plotted on a map.

“Rebalancing the Investment in Collections,” discussed that libraries had painted themselves into a corner by focusing too much on their collection budgets. Investing in the right skills and partnerships is most critical now. “The comprehensive and well-crafted collection is no longer an end in itself.”

On person told librarians that they shouldn’t rush to be the first to digitize everything and invest in every new technology. “Everybody underestimates the cost of innovation,” he said. “Instead of rushing in and participating in a game where you don’t have the muscle, you want to stand back” and wait for the right moment.

Digital Preservation Matters.

Digital Preservation-Friendly File Formats for Scanned Images.

Digital Preservation-Friendly File Formats for Scanned Images.  Bill LeFurgy. The Signal. October 12, 2011.

Some digital file formats are better for preservation than others.  The best format for preservation is one where the content can be viewable accurately regardless of changes in hardware, software or other technical changes. The Library of Congress has created a web resource to help in selecting file formats, and which will help in understanding how effective formats for long-term preservation.

• Disclosure of specifications and tools for validating technical integrity
• Adoption by the primary creators and users of information resources
• Openness to direct basic and non-propriety tools
• Self-documentation of metadata needed to render the data as usable information or understand its context
• Degree to which the format depends on specific hardware, operating system, or software for rendering the information and how difficult that may be.
• Extent that licenses or patents may inhibit the ability to sustain content.
• Technical protection mechanisms. Embedded capabilities to restrict use in order to protect the intellectual property.

 Using these factors has helped determine formats that may be more sustainable than others. 

To Save and Project Fest: Long Live Cinema!

To Save and Project Fest: Long Live Cinema!  J. Hoberman. The Village Voice. October 12, 2011.
Digital might be the future of the motion-picture medium, but for film preservation, it’s a mixed blessing. Archivists make it clear that digital technology is part of the solution—and part of the problem. Digital cinema is itself difficult to preserve, subtly distorts (by “improving”) the celluloid image, even as it often dictates (through commercial considerations) those movies deemed worthy of preservation. New York Times DVD critic Dave Kehr has pointed out that instead of increasing access, each new distribution platform (from 35mm to 16mm, VHS, DVD, Blu-ray, and online streaming) has narrowed the range of titles in active distribution and diminished the proportion of available films. Film restoration is also the restoration of cultural memory.