Digital documents debut on historically endangered list.

Digital documents debut on historically endangered list. MaineBiz. August 30, 2012.

Digital documents debut on historically endangered list for the first time, Maine's most endangered historical assets now include digital records, according to an annual survey by the nonprofit Maine Preservation. This was done to emphasize the importance of the preservation of these documents to others. Electronic documents are in particular danger because of the ease with which they can be lost or destroyed. "We've not figured out how to manage this material. There's such a proliferation that nobody thinks about this material."
 

Digital Preservation Matters.

The Problem of Data

The Problem of Data. Lori Jahnke, Andrew Asherpub, Spencer D. C. Keralis. CLIR Report. Council on Library and Information Resources. August 12, 2012.
Excellent report on data storage, use, and curation.  A section contains a snapshot of the current digital data curation education landscape.  Below are some long notes and excerpts from the PDF article:

Key Findings

• None of the researchers interviewed for this study have received formal training in data management practices, nor do they expresssatisfaction with their level of expertise.
• Few researchers, especially among those who are early in their career, think about long-term preservation of their data.
• The demands of publication output overwhelm long-term considerations of data curation. Metadata and documentation are of interest only if they help a researcher complete his or her work.
• There is a great need for more effective collaboration tools, as well as online spaces that support the volume of data generated and provide appropriate privacy and access controls.
• Few researchers are aware of the data services that the library might be able to provide and seem to regard the library as a dispensary of goods (e.g., books, articles) rather than a place for research/professional support.

Recommendations

• There is unlikely to be a single out-of-the-box solution that can be applied to the problem of data curation. Instead, an approach is needed that emphasizes working with researchers to identify or build appropriate tools.
• Researchers must have access to adequate networked storage.
• Universities should revise access policies to support multi - institutional research projects.
• Programs should begin early in the researcher career path for the greatest long-term benefit.
• Data curation systems should be integrated with the active research phase (i.e., as a backup, etc).
• Privacy and data access control tools should be developed to manage confidential data. Policies must be developed that support researchers in using these technologies.

Other notes:

• Data curation, a term generally defined as a set of activities that includes the preserving, maintaining, archiving, and depositing of data to keep it secure, intact, and accessible for reuse.
• Many researchers expressed concerns surrounding the ethical reuse of research data. Additional work is needed to establish best practices in this area, particularly for qualitative data sets.
• Most participants reported feeling adrift when establishing protocols for managing their data and added that they lacked the resources to determine best practices, let alone to implement them. Almost none of the scholars reported that data curation training was part of their graduate curriculum.
• Perhaps one of the more complicated issues for data curation is the complex life cycle of research data and projects. Data collection may occur throughout the project and change from before it is completed.
• Scholars may collect data on a phenomenon unrelated to their current project with no clear idea of the potential usefulness of those data. Such data might be integrated with a later project, given away to an interested colleague, or never used at all.
• It would be helpful to have a way to collect data into a collection space that could be used throughout the project.
• The researchers held contradictory views about the value of their data. Some wanted to associate their data with publications or to have it available for use in the classroom
• Few of the researchers thought about long-term preservation of their data, especially those who were early in their career.
• The academic system offers little or no career reward for preserving one’s data.
• Data preservation strategies must take into account varied, proprietary, and non-standard data formats, and provide a real-time benefit for the scholar in meeting research goals.
• Given the lack of infrastructure for sharing and storing data, the social sciences may face similar problems of data loss in documenting social phenomena as researchers begin to work within larger collaborative groups and with larger data sets. Data stored on personal media devices are especially vulnerable to this type of loss, as few scholars have the skills necessary to maintain data over time and across hardware and software platforms. Several of the scholars interviewed reported storing data on legacy systems that may become inaccessible
• University policies that appropriately address the ethical considerations relating to data sharing and preservation would benefit researchers, administrators, and technologists alike.
• Researchers hold tremendous amounts of data on personal computers and hard drives, many of which are not backed up adequately. Among the participants, the research data ranged from under 1 GB to multiple terabytes. Data types included various formats of images, video, audio files, data sets, documents, etc.
• Managing large files presents significant challenges for researchers in that university infrastructures typically do not provide adequate storage space or sufficient bandwidth for data access.  The data may be lost when researchers upgrade their computers or software. Few researchers put more than minimal effort into organizing non-active data or ensuring its continued compatibility with new software or hardware.
• There is a clear need for libraries to move beyond passively providing technology to embrace the changes in scholarly production that emerging technologies have brought.  
• The data preservation step must be fully integrated into a scholar’s research workflow. Not only are necessary metadata and other materials much more easily captured while research is in progress, but also there is a real opportunity to streamline research workflows and to provide much needed support. Scholars need help with the technical aspects of managing and preserving data, as well as with basic curation issues (e.g., what to keep and what to delete), and the ethical implications of sharing their data (e.g., what is an appropriate latency period for the data and how does one balance the need to provide meaningful access with the risk of inadvertently exposing confidential participant information).
• Although some researchers acknowledge that their data could be useful to other researchers, there is little incentive to invest time in archiving or repackaging data sets.
  
• Extensive outreach to scholars is necessary to build the relationships that will facilitate data preservation. This is likely to be a slow process initially. Researchers are unlikely to engage with those they do not view as peers.
  
• Researchers need additional tools to manage preserved data on their own, and they would benefit from access to professionals who can offer advice on management strategies.
• Researchers typically align themselves with their disciplines rather than with their institutions; therefore, support models that extend beyond the university are likely to be especially beneficial.
• Reaching the level of collaboration among universities and the technical interoperability required to capture and preserve a career’s worth of data in the current environment is a challenge.
• Current data management systems must be fundamentally improved so that they can meet the capacity demand for secure storage and transmission of research data. Integrating the data preservation system with the active research cycle is essential to encourage researcher investment.
• Researchers are not well positioned to meet the technical and policy challenges without the coordinated support of libraries, information technology units, and professionals who possess both technical and research expertise.

•  One example concerning the PETRA e+e collider project in Hamburg, Germany; In the more than 25 years since, theoretical insights and computing advancements have made the data valuable once again. However, much of the data have been irrevocably lost to corrupt storage media, lost computer code, and deactivated personal accounts. These early particle physics experiments are unique, as modern colliders operate at higher energy levels and cannot replicate the particle interactions.

Digital Preservation Matters.

OAIS 2012 update.

OAIS 2012 update. Barbara Sierman. juli 30, 2012.

Reference Model for an Open Archival Information System (OAIS). Magenta Book. Issue 2. June 2012. 

The main changes are the following:

·         Access Rights information is added as an element to the Preservation Description Information.

·         emulation as a strategy to preserve access services or the original look and feel and explains how different varieties of emulation will fit in the OAIS model.

·         There is more interaction between the Administration Functional Entity and the Preservation Planning Functional Entity. The Preservation Planning Functional entity will create preservation plans (and not only “migration plans” as mentioned in the previous version), based on its monitoring activity,  and will send these to the Administration functional entity to be performed. But the Administration Functional Entity (cq.  the function The Establish Standards and Policies) will also receive periodic risk analyses created by Preservation Planning to act upon, which gives the Preservation Planning Functional Entity a more active role in monitoring not only the outside world but also the OAIS itself.  Interaction also takes place when Preservation Planning sends recommendations on AIP updates, and The Administration Functional Entity replies with preservation requirements (added to the already exisiting “migration goals and approved standards”). So for creating new migration packages not only the preservation requirements resulting from monitoring the Designated Community are input, but also the preservation requirements from Administration. In general some loose ends seems to be united here.

·         The confusing use of the word “authentication” has now changed and the term “authenticity” is defined (adding the much used definition of “The degree to which a person (or system) regards an object as what it is purported to be.” With an important addition: “ Authenticity is judged on the basis of evidence.”

·         The term Information Package is redefined to : “ A logical container composed of optional Content Information and optional associated Preservation Description Information. Associated with this Information Package is Packaging Information used to delimit and identify the Content Information and Package Description information used to facilitate searches for the Content Information.”

·         A new definition is introduced “Other Representation Information” and described as “Representation Information which cannot easily be classified as Semantic or Structural. For example software, algorithms, encryption, written instructions and many other things may be needed to understand the Content Data Object, all of which therefore would be, by definition, Representation Information, yet would not obviously be either Structure or Semantics. Information defining how the Structure and the Semantic Information relate to each other, or software needed to process a database file would also be regarded as Other Representation Information.”

·         This chapter is adapted with the above mentioned changes and refines the definitions for an AIP version (An AIP whose Content Information or Preservation Description Information has undergone a Transformation on a source AIP and is a candidate to replace the source AIP. An AIP version is considered to be the result of a Digital Migration.) versus an AIP edition (An AIP whose Content Information or Preservation Description Information has been upgraded or improved with the intent not to preserve information, but to increase or improve it. An AIP edition is not considered to be the result of a Migration.)

 

Digital Preservation Matters.

Bit Rot & Long Term Access.

Bit Rot & Long Term Access. Bram van der werf.  Open Planets Foundation. 28 February 2011.
Some organizations are looking at bit rot in image collections.  Open Planets wiki has a Problem, Requirement, Use Case sequence.  "We need to make serious efforts to drill down into every individual potential element and risk for Bit Rot. Create Use Cases and take action on these Use Cases and develop practical tools and/or workflow to assure we can prevent Bit Rot." It may involve "big development projects and often more a case of creating scripts that can compare "checksums" or "open and save as new" type of activities inside a repository. Todays hardware is subject to a very short technical life cycle and by far not as reliable for keeping the integrity of the bits as we sometimes like to believe." The site also includes a video of Vint Cerf's view on Bit Rot.

Digital Preservation Matters.

An interactive eGuide: Backup Solutions

 An interactive eGuide: Backup Solutions. Computerworld. June 2012. [PDF]

Disaster recovery is the ability to continue your mission-critical operations after an interruption of some kind. An organization must be able to restore applications and processes to the point where they were before the outage occurred. Organizations “with a business continuity plan and disaster recovery plan have the ability to get back to a semblance of normalcy in a much shorter span of time than those without.”

Instead of focusing on technology,  put disaster recovery processes in place; prepare for the most likely causes of downtime. Decide what is most important. Store your data away from your physical site in the event of a natural disaster.  Develop strategies to address operational continuity,  IT recovery, and communication needs.   Understand all parts of the backup procedures.

• Test your backup/recovery (including cloud backup) at least one a year and make sure it actually works the way you want it to.
• Keep install files for your software, including security keys.

“Backup is really the insurance policy that you hope you never have to cash.

Digital Preservation Matters.

WARCreate Google Chrome Extension. “Create WARC files from any webpage”.

WARCreate Google Chrome Extension. “Create WARC files from any webpage”. Mat Kelly. Web page. June 2012.
WARCreate is a Google Chrome extension that allows a user to create a Web ARChive (WARC) file from any browseable webpage. The resulting files can then be used with other tools like the Internet Archive's open source Wayback Machine. The tool is an evolving product with the end result pushing toward being a personal web archiving solution for those that wish to securely archive their metadata in a standardize way. This tools is still under development.

Digital Preservation Matters.

New cloud service safeguards organisations’ digital assets.

New cloud service safeguards organisations’ digital assets. Press Release. 27 June 2012. Tessella has announced Preservica, a new cloud-based service that provides organisations with a secure and affordable solution to safeguard their digital assets. Preservica runs on Amazon Web Services (AWS). It is built on Tessella’s SDB technology and provides tools required to build a long-term digital preservation solution including:

•     Ingest – upload digital content and metadata from a variety of content sources
•     Content storage – encrypt and store multiple copies in a safe, backed-up location
•     Flexible metadata and security – choose how is arranged, described, and protected
•     Access – full search, browse, and download facilities via browser interface
•     Preservation tools – ensure that content is protected against future obsolescence.

Digital Preservation Matters.

 

The Seven Deadly Sins of Records Retention

The Seven Deadly Sins of Records Retention.  Sarah D. Scalet. CSO Security and Risk.  July 01, 2006.

 Legal actions in the past few years have made document retention programs important. One wrong step can cost an organization money.  Some have concluded that they should archive, forever, anything and everything to be on the safe side. But keeping too much information is a risk too, as you can expose yourself to litigation risks, and possibly violating privacy rights. 

1.       Not keeping your records straight from your backup.

• The first step to a good records management program is simply identifying what a record is. E-mail servers and network drives get backed up to keep the business running. But a record is "something that you need to keep around for a set period of time, either for regulatory, legal or business reasons. Records encompass both structured information, like financial transactions ... and unstructured information, like financial spreadsheets." 
• "while backup media may be in a continual state of being written and overwritten, records that must legally be retained often need to be stored on immutable, nonrewritable storage, and should be either very well-organized, very easily searched or both.

2.       Expecting the legal department to produce a rule of thumb for how long to store records.

3.       Assuming that document retention is someone else's job.

4.       Not being able to respond quickly to a request.

5.       Having a policy you can't follow.

6.       Failing to offer guidance on how to destroy old records.

7.       Telling people to delete information at the wrong time.

Start with an accurate survey of the information that's in the organization, a data map.
"At the end of the day, you have to have some sort of written policy around it."

Digital Preservation Matters.

Online Sharing of Digital Preservation Problems and Solutions

Online Sharing of Digital Preservation Problems and Solutions. bram van der werf. Open Planets Foundation. 27 October 2011.
The vision of the Open Planets Foundation from the beginning is that meaningful discussions around actual Digital Preservation problems are key to finding solutions, tools and ultimately improving digital preservation practice.  It encourages people to document issues and search for practical solutions to the problems. 

The online digital preservation community needs to be driven by content issues, practical solutions and above all meaningful discussions between all stakeholders. Digital Preservation might be a technology challenge but it is definitely not for the sake technology, it is about keeping digital content safe and accessible. The technology serves the digital content users and curators. The digital preservation practitioners for small to medium institutions where people have multiple roles are clearly the collection owner. For larger institutions, this becomes more of a distributed role among those with different technical responsibilities. By involving all the groups everyone can benefit from the larger organizations and  can often take actions on today’s problems as experienced by practitioners of medium/small institutes.

Digital Preservation Matters.

Metadata Clean Sweep: A Digital Library Audit Project

Metadata Clean Sweep: A Digital Library Audit Project. R. Niccole Westbrook, et al.
As digital library collections grow in size, metadata issues such as inconsistencies, incompleteness and quality become increasingly difficult to manage over time.  This was a project by library school interns and full-time staff to alleviate poor recall, poor precision and metadata inconsistencies across digital collections. Access to content is the primary goal of the metadata of a digital library. Successful metadata maintenance must include both correction of found mistakes and correction to practices that lead to those mistakes.

Digital Preservation Matters.

A simple JP2 file structure checker.

A simple JP2 file structure checker. Johan van der Knijff. Open Planets. 1 September 2011.
One major risk of file migration is that hardware failures during the migration process may result in corrupted images. Ideally JHOVE would be able to detect such errors using format validation tools.  "I started out with removing some trailing bytes from a lossily compressed JP2 image. As it turned out, I could remove most of the image code stream (reducing the original 2 MB image to a mere 4 kilobytes!), but JHOVE would still say the file was "well-formed and valid". I was also able to open and render these files with viewer applications such as Adobe Photoshop, Kakadu's viewer and Irfanview. The behaviour of the viewer apps isn't really a surprise, since the ability to render an image without having to load the entire code stream is actually one of the features that make JPEG 2000 so interesting for many access applications. JHOVE's behaviour was a bit more surprising, and perhaps slightly worrying." So he wrote a very simple JP2 file structure checker that is available for download.

Digital Preservation Matters.

 

The CLIF Project: The Repository as Part of a Content Lifecycle

The CLIF Project: The Repository as Part of a Content Lifecycle. Richard Green, Chris Awre, Simon Waddington. Ariadne. 9 March 2012.
This was a joint project that did an extensive literature review and worked with  digital content creators to understand how to deal with the interaction of the authoring, collaboration and delivery of materials. At the heart of meeting institutional requirements for managing digital content is the need to understand the different operations through which content goes, from planning and creation through to disposal or preservation. Repositories must be integrated with the other systems that support other parts of this lifecycle to prevent them becoming yet another information silo within the institution.

The CLIF software has been designed to try and allow the maximum flexibility in how and when users can transfer material from one system to another, integrating the tools in such a way that they seem to be natural extensions of the basic systems.  This open source software is available for others to investigate and use.

The repository’s archival capability is regarded as one of its strongest assets, and the role of the repository within a University will be regarded very much in terms of what it can offer that other campus systems cannot.  It should not try to compete on all levels. There is a need to clarify better at an institutional level what functionality is offered by different content management systems, in order to better understand how different stages of the digital content lifecycle can be best enabled.

Digital Preservation Matters.

Pogoplug service turns your computers into private cloud

Pogoplug service turns your computers into private cloud. Lucas Mearian. Computerworld. May 9, 2012.
Pogoplug Team which allows home offices and small businesses (best for 10 - 50 people) to turn file servers or spare PCs into pools of storage accessible from the Web. This basically creates a private cloud for businesses so they can store information within their own firewalls rather than on third-party services. The product allows multiple users to share storage in a home or office. The service's capabilities are similar to Dropbox for file sharing and collaboration. It has full access and file sharing to 3TB of storage for $150 (not including the $15-a-year, per-person licensing fee paid to Pogoplug for its service.)

A prototype JP2 validator and properties extractor.

A prototype JP2 validator and properties extractor. Johan van der Knijff. The Open Planets Foundation.  14 December 2011, Update March 2012.   

The site includes a discussion of the full-fledged JP2 validator tool, as well as how it validates the object. A prototype is now ready in the form of the jpylyzer tool. The tool is both a validator and a properties extractor.

Digital Preservation Matters.