PREMIS Data Dictionary for Preservation Metadata, Version 3.0

PREMIS Data Dictionary for Preservation Metadata, Version 3.0. Library of congress.
June 10, 2015. [Full PDF]
The PREMIS Data Dictionary and its supporting documentation is a comprehensive, practical resource for implementing preservation metadata in digital archiving systems. The Data Dictionary is built on a data model that defines five entities: Intellectual Entities, Objects, Events, Rights, and Agents. Each semantic unit defined in the Data Dictionary is a property of one of the entities in the data model.

This new publications are:

• PREMIS Data Dictionary. Version 3.0. This is the full document which includes the PREMIS Introduction, the Data Dictionary, Special Topics, and Glossary.
• PREMIS Data Dictionary This document only has the Data Dictionary, introductory materials
• Hierarchical Listing of Semantic Units: PREMIS Data Dictionary, Version 3.0
• The Version 3.0 PREMIS Schema is not yet available

Version 3 of the Data Dictionary includes some major changes and additions to the Dictionary, which are:

• Reposition Intellectual Entity as a category of Object to enable additional description within PREMIS and linking to related PREMIS entities.
• Reposition Environments (i.e. hardware and software needed to use digital objects) so that they can be described and preserved reusing the Object entity. That is to say, they can be described as Intellectual Entities and preserved as Representation, File or Bitstream Objects.
• Add physical Objects to the scope of PREMIS so that they can be described and related to digital objects.
• Add a a new semantic unit to the Object entity: preservationLevelType (O, NR) to indicate the type of preservation functions expected to be applied to the object for the given preservation level.
• Add a new semantic unit to the Agent entity to express the version of software Agents: agentVersion (O, NR).
• Add a new semantic unit to the Event entity: eventDetailInformation (O, R)

There are major additions in the “PREMIS Data Model” and “Environment” sections.
The data model:


The entities in the PREMIS data model are:

• Object: a unit subject to digital preservation.This can now be an environment.
• Environment: technology supporting a Digital Object. Can now be as Intellectual Entity.
• Event: an action concerning an Object or Agent associated with the preservation repository.
• Agent: entity associated with Rights, Events, or an environment Object.
• Rights Statement: Rights or permissions pertaining to an Object and/or Agent.

With the advent of Intellectual Entities in PREMIS 3.0, environments have been transformed. "Before version 3.0, there was an environment container within an Object that described the environment supporting that Object. If a non-environment Object needs to refer to an environment, it is now recommended that the environment is described as an Object in its own right and the two Objects are linked with a dependency relationship."

Digital Preservation Matters.

30 percent of servers are ‘Comatose’, capital in enterprise data centers is wasted

New data supports finding that 30 percent of servers are ‘Comatose’, indicating that nearly a third of capital in enterprise data centers is wasted. Jonathan Koomey and Jon Taylor. Anthesis Group. June 3, 2015.
The report shows that utilization of servers in business and enterprise data centers “rarely exceeds 6%” (i.e, they deliver no more than six percent of their maximum computing output on average over the course of the year) and two groups have found that up to 3.6 million servers (30%) are comatose, meaning that they are using electricity but delivering no useful information services (abandoned or never used).

"In the twenty first century, every company is an IT company, but too many enterprises settle for vast inefficiencies in their IT infrastructure. The existence of so many comatose servers is a clear indication that the ways IT resources in enterprises are designed, built, provisioned, and operated need to change. The needed changes are not primarily technical, but revolve instead around management practices, information flows, and incentives." The two groups offer online efficiency calculators.

Digital Preservation Matters.

Digital Preservation Metadata and Improvements to PREMIS in Version 3.0

Digital Preservation Metadata  and Improvements to PREMIS in  Version  3.0. Angela Dappert. May 27, 2015. [PDF]
This is the notes from a DCMI/ASIS&T joint webinar about PREMIS v. 3. The PDF document has 63 slides which gives an overview of why digital preservation metadata is needed, shows examples of digital preservation metadata, shows how PREMIS can be used to capture this metadata, and shows some of the changes in version 3.0.

Digital preservation metadata is the metadata needed to ensure long-term accessibility of digital resources. Digital objects must be self-descriptive independently from the systems that were used to create them. PREMIS is the de-facto standard for metadata to support the preservation of digital objects and ensure their long-term usability. It is a common data model for organizing/thinking about preservation metadata, or for exchanging information packages between repositories. It is not an out-of-the-box solution, nor all the metadata needed.

Digital Preservation Matters.

Funding for preservation software development

Funding for preservation software development. Gary McGath. File Formats Blog.  June 9, 2015.
The Open Preservation Foundation is launching a new model for funding the development of preservation-related software. The model allows organisations to support individual digital preservation software products and ensure their ongoing sustainability and maintenance. "US libraries have been rather insular in their approach to software development. They’ll use free software if it’s available, but they aren’t inclined to help fund it. If they could each set aside some money for this purpose, it would help assure the continued creation and maintenance of the open source software which is important to their mission."

Digital Preservation Matters.

File identification tools, part 5: JHOVE

File identification tools, part 5: JHOVE. Gary McGath. File Formats Blog. June 11, 2015.
JHOVE is a tool that identifies and validates AIFF, GIF, HTML, JPEG, JPEG2000, PDF, TIFF, WAV, XML, ASCII, and UTF-8 files. Unrecognized files are called a “Bytestream.” 

Key concepts in JHOVE are “well-formed” and “valid.” A file which is “well-formed but not valid” has errors, but not ones that should prevent rendering. JHOVE focuses on the semantics of a file rather than its content.  It only reports full conformance to a profile. It won’t tell you why it fell short.

Download JHOVE from GitHub the Open Preservation Foundation; (do not download from SourceForge). Documentation is on the OPF website. A developer's guide is also available:   JHOVE Tips for Developers.

It shouldn't be confused with JHOVE2 which does similar things but has a different code base.

Digital Preservation Matters.

File identification tools, part 4: ExifTool

File identification tools, part 4: ExifTool. Gary McGath. File Formats Blog.  June 8, 2015.
The ExifTool, which analyzes Exif metadata, is an open source library and command line tool for identifying, editing, and extracting metadata from many formats, such as image, movie, and audio files. The command options can return metadata in a variety of formats or edit files.

Digital Preservation Matters.

IFI Irish Film Archive publishes new Digital Preservation and Access Strategy

IFI Irish Film Archive publishes new Digital Preservation and Access Strategy. Seán Brosnan. Irish Film and Television Network. 16 Jun 2015. [PDF]
The Irish Film Archive has published a new Digital Preservation & Access Strategy that addresses archiving vast quantities of moving image material in a digital environment.  “This crucial document ... outlines a long-term plan and a set of guiding principles, flexibility, scalability and sustainability, which will assist us in preserving and providing access to Ireland’s digital moving image material.”

In order to fully achieve its mission, the organization must be able to preserve and make available the digital collections in its care, by doing the following items:

• Publish a comprehensive Digital Preservation and Access Strategy document.
• Upgrade the Archive’s technology and equipment to facilitate digital activities.
• Develop policies and procedures to support the digital strategy and asset management
• Develop a trusted digital repository
• Increase access to the collections
• Secure funding to ensure digital preservation and access are supported and sustainable.
• Develop specialist digital management and preservation skills
• Develop the Archive’s existing database to allow the recording necessary metadata
• Follow international examples of best practice
• Create and maintain on-going partnerships

The digital strategies and solutions must ensure efficiency and cost effectiveness by being flexible, scalable, and sustainable.  The Digital Preservation and Access Strategy must consider the short-, medium-and long-term needs of the organization in safeguarding and sharing digital collections. The risks and challenges they face include:

• Technical obsolescence
• Lack of Standards
• Expense: Creating and maintaining the infrastructure, equipment and trained people will be an ongoing, costly and resource-heavy activity, but it is vital to ensure the availability of digital collections to future generations. An efficient digital preservation strategy "cannot be achieved with sporadic investment, but must be supported by regular and predictable funding"

The document also describes benefits, opportunities, stakeholders and strategic priorities. The strategic objectives include:

• To maintain the Archive to the highest international standards
• To improve resources within the Archive
• To  improve access to the collections
• To heighten awareness of the collections for current and future generations

Digital Preservation & Access Strategy Document will be an essential tool in helping the organization become a Digital Archive that meets internationally recognized standards of excellence. This will be a living document and will reviewed regularly to ensure it still meets the strategic objectives.  It will take into account any changes in access or preservation technology, policies, and budget changes. The document will also be supplemented by additional policies over time.

Digital Preservation Matters.

How Do Users Search and Discover?

How Do Users Search and Discover? Christine Stohn. Ex Libris. May 2015.
This paper describes the findings of a recent Ex Libris user study and discusses how the conclusions apply to library discovery systems. The challenging goal for libraries is to enable libraries to bring together users, their research intent and needs, and the wealth of available information. Intelligence about users’ behavior, intent, and expectations in the context of the tasks that the users strive to accomplish is of key importance in endeavoring to achieve this goal. User studies are not the same as usability studies. The core functionality of a discovery system is search and find, but different users have different needs.

Digital Preservation Matters.

Digital Preservation links

Digital Preservation links . Julie C. Swierczek. scholar.harvard.edu website. June 2015.
A very thorough and up to date list of digital preservation resources on the internet. There are many topics, such as:

• Digitization
• Digital Forensics
• Digital Accessioning Workflows
• Guidelines/best practices
• Reports/whitepapers/publications/bibliographies/journals
• Metadata, vocabularies, XML and harvesting
• Standards
• Projects of interest
• Blogs
• Web archiving

 A very useful reference resource.

Digital Preservation Matters.

Digital Preservation 101, or, How to Keep Bits for Centuries

Digital Preservation 101, or, How to Keep Bits for Centuries. Julie C. Swierczek. scholar.harvard.edu website. June 4, 2015.
An interesting presentation that looks at what digital archivists need to preserve electronic records permanently. It discusses:

• OAIS model for long-term digital preservation;
• requirements of a trustworthy digital repository;
• preferred file formats for long-term storage; 
• digital forensics and FRED machines;
• definitions and differences of “archives” and “backup copies”.

Digital Preservation Matters.

Preserving the Born-Digital Record: Many more questions than answers

Preserving the Born-Digital Record: Many more questions than answers. James G. Neal. American Libraries. May 28, 2015.
The world is producing vast amounts of born-digital material. The volume, complexity, and dynamism of this information challenge us to think creatively about its capture, organization, and long-term preservation and usability. What is the role of the library? Is this a source of failure or opportunity for the global library community?

This is an issue of integrity, of the collective adherence to a code and standard of values, of maintaining human records as complete, unimpaired, and undivided as possible. The ability to consult the evidence and sources used by researchers and authors will be lost if those digital records are not available. The ability to research and investigate the history and current state of our world will be compromised if born-digital materials are gone or changed. The ability to access the sources of record will be difficult if they are deposited and dispersed into multiple and disparate sites. This is the challenge of repository chaos.

At the core of born-digital content preservation and archiving are four principles.

1. We cannot preserve what we have not collected.
2. We must enable access, which brings persistence.
3. We must secure and curate the content.
4. We must take care of the content as steward.

How does born-digital content fit into what libraries do? Libraries select, acquire, synthesize, preserve and archive information, and enable users to understand, use, and apply information. This supports teaching and learning, research and scholarship.

Quality equals content plus functionality. To make sure that the born-digital content is preserved and usable in the long term we must understand and accommodate the important characteristics of digital information. With born-digital resources we must also consider the relationship among form, text, and function, context, renderability, and versioning over time. "We see the inevitability of physical and format obsolescence, the importance of authenticity and provenance, and the role of standards such as globally unique identifiers."

The scope, depth, and cost of the threat mean that individual libraries adequately preserve born-digital content alone. We need to promote cooperation and new public–private partnerships. The Digital Preservation Network (DPN) is an example of this. "We will not have the technologies, tools, workflows, or standards unless we work together in new ways."

Libraries must take on responsibility for the preservation of born-digital content.

Digital Preservation Matters.

TIFF/A Standard Initiative

TIFF/A Standard Initiative. Website. June, 2015.
The TIFF/A standard initiative intends to create an ISO specification of a Archival TIFF Format. TIFF is a widely used format, but it is complex and has some features not suited for long term preservation. The TIFF/A-specification will be enhanced with mandatory and forbidden tags for archival purposes, similar to PDF/A. "This standard will be created in parallel with DPF Manager, an open source TIFF format validator that, in addition to the current TIFF ISO Standards, will be the first conformance checker for the TIFF/A new standard." The group looks to create a community of experts interested in discussing the initiative in order to prepare a proposal to submit to the ISO.

Digital Preservation Matters.

File identification tools, part 3: DROID and PRONOM

File identification tools, part 3: DROID and PRONOM. Gary McGath. File Formats Blog.  June 1, 2015.
DROID (Digital Record Object IDentification) is an open sourced Java-based tool from the UK National Archives that is designed to identify and verify files for digital repositories.  It relies on file format information from the National Archive’s registry, which uses a tool called PRONOM. "DROID depends on files that describe distinctive data values for each format". It can verify single files or large batches of files, or it can be integrated into other applications. DROID generates reports about the file and the identify and verification, or report if it can't match the type of file. Sometimes it may report that a file has more than one matching signature, such as if there is more than one version of a format.

Digital Preservation Matters.

What We've Saved (2004-2014)

What We've Saved (2004-2014). Andy Jackson. UK Web Archive. June 11, 2015. [PPT slides]
After 10 years of the UK web archive, what has been saved? Three collections, over 8 billion resources, and 160 TB of compressed data. "Looking inward is not enough: To understand the value of our collection, we need to look beyond our walls and put it in context." A review shows how much has been lost from the web. Almost 100% of the crawled urls in the UK web archive, are gone or missing on the internet. And about 40% from 2013 is gone or missing. Link rot & content drift dominate:

• 50% of resources unrecognisable or gone after 1 year
• 60% after 2 years, 65% after 3 years (islands of stability)
• Noticeably higher rot rate than results for legal/academic web

Simple similarity measures provides some insights, but there needs to be more work to look for old content in new locations.

Digital Preservation Matters.

Libraries and Research Data Services

Libraries and Research Data Services. Megan Bresnahan, Andrew Johnson. University of Colorado Boulder. 2014. [PDF]
A presentation that looks at the importance of training librarians to become experts in research data services (RDS). “Reassigning existing library staff is the most common tactic for offering RDS. This approach also needs to be supported with professional development for staff so they can gain the required expertise to provide the full range of RDS”

Some feedback from Subject Librarians shows that they know it is part of their duties and that it is becoming more important, but that it is difficult to accomplish in the current environment:

• “Research data is intimidating!”
• “How can I take on research data support with so much else already on my plate?!”
• “I need practical tools to use to help researchers with their data”
• “Helping faculty and students with their data is an increasingly important part of my liaison duties”

Becoming expert requires data services training with established learning goals. With training, Subject Librarians are able to:

• Understand the stages
• Define the role
• Apply skills
• Plan for outreach
• Feel confident
• Engage with researchers

DataQ is a website that serves as a collaborative, peer-reviewed reference tool for librarians providing research data services

Digital Preservation Matters.

National Library of the Netherlands-Portico Partnership at the Forefront of Digital Preservation and International Collaboration

National Library of the Netherlands-Portico Partnership at the Forefront of Digital Preservation and International Collaboration. Portico Press release. 28 May 2015.  
The National Library of the Netherlands (KB) and Portico created a new partnership that will  preserve e-journals through the KB’s e-Depot program, which preserves locally published content in the Netherlands. The program focuses on the preservation of e-journals from international scientific publishers and  “At the KB, our preservation strategy has been to use a variety of solutions and tools, and not rely on any single method.” Portico has been a key collaborator in their preservation work over the years.

The e-Depot already preserves more than 15 million e-journal articles. Portico will provide the KB with preservation-formatted e-journal content from scientific publishers who are new to the program including BioOne, Walter de Gruyter, Wolters Kluwer, Karger, Brill, and Thieme. An e-journal trigger event would allow the KB would provide journal access to Dutch researchers.

Digital Preservation Matters.

FFmpeg's FFV1 lossless video codec: A Free Software success story

FFmpeg's FFV1 lossless video codec: A Free Software success story. Peter Bubestinger. 9 May, 2015. Published 2 June 2015.
The Österreichische Mediathek wanted to digitize video for long-term preservation and looked at the options. For audio formats they used uncompressed /lossless PCM (WAV). They expected to apply the same requirements to video, but uncompressed video was not an option because the file size was too large.Majority of the institutions preserving video use lossy codecs:

• IMX, ProRes, MPEG-2, MPEG-4
• Industry proposed format:  JPEG2000-lossless/PCM in MXF

They looked at FFV1 as a preservation codec (it originated as FFmpeg), and started using it in house in 2011, and began working with the implementation:

• Double checked FFV1's long-term sustainability
• Tested its implementation
• Documented its usage
• Organized and funded improvement

They solved their problem of digitizing and archiving video and the resulting solution is free to use. 
The  FFV1 Video Codec Specification by Michael Niedermayer.

Lessons learned:

• User perception is of great importance
• Provide graphical tools
• No one wants to be the first to try something out
• The look of a website is important. A "cheap looking website" gives an impression of a low quality product
• Provide documentation
• Make access easy
• Provide option of paid setup and support

Digital Preservation Matters.

New Sources and Storage Options For Rosetta

Rosetta Users Group 2015: New Sources and Storage Options For Rosetta. Chris Erickson. June 3, 2015. [PDF slides]
This is my presentation at the Rosetta's User Group / Advisory Group held this past week.

We installed Rosetta in March 2012 and have ingested a number of collections in the preservation repository. In addition to those sources and collections we have already set up to work with Rosetta, we have been working with some new areas. These include:

• University Academic electronic records in SharePoint. Rosetta harvest tool for SharePoint
• New Library repositories, such as Digital Commons
• Harvest Canon camera raw images and ingest into Rosetta
• Ingest University videos and digitized files from the Audio Digitization project
• Program to gather information from Unstructured folders of archival objects and ingest into Rosetta

One of the things that everyone is struggling with is the lack of sufficient storage. Conventional storage is expensive and limited. So we are investigating alternative long term storage possibilities:

• Develop a Proof of Concept Project to ingest Rosetta content in DPN
• Amazon S3 Cloud Storage from Rosetta. Easy to connect to Rosetta.
• Hitachi - LG Data Storage (HLDS) Optical Archive System
• Long Term storage system 
• Lowest storage cost
• Single rack unit holds 1 PB of permanent storage; unlimited expansion
• Currently testing with our Rosetta system
• Reduce the need for refreshing or migrating content
• Plan to use with Millenniata M-Discs in the archive

What we are looking for in preservation storage:

• Sufficient capacity for our ever increasing content
• Reasonable long term cost
• Lower total costs of ownership
• Reduced cost of refreshing or migration
• Reliable and recoverable
• Archival media
• Industry Storage Partner
• Multiple copies, locations
• Secure storage
• Accessible
• Rosetta
• Network

Digital Preservation Matters.

Writing in the Sand

Writing in the Sand. Chris Erickson. June 4, 2015.
Recently I have been sorting through some family photographs from long ago. One image, digitally born, was taken on a family holiday at a beach. In the sand we had written a birthday greeting to our son. The greeting, which only lasted a short time, was erased by the wind and waves. The snapshot is all that is left of the original message.

It reminded me of the problems we now face regarding digital content.   Like writing on the beach, our digital data is based on electric or magnetic charges on wafers of sand. The messages are likewise temporary,  erased before long by the elements.

There are lots of problems encountered in preserving digital content, but it seems the first problem is that we entrust our permanent messages to temporary media. We then rely on complex procedures to lengthen the life of the temporary media. Or wring our hands when the media fail. What do we expect when we use magnetic based media while sitting on the world's largest magnet?

What will help? I suggest that we change our mind set and stop writing our precious, permanent  content to temporary media.  (And I am not suggesting that we go towards unique one-of solutions that must be created in a lab at enormous cost.) There are few options that can be used, but if we decide to make this our goal there will be more innovation in the area.  One option that does exist right now is the M-Disc technology. It was designed at our university with permanent storage in mind; cost effective digital storage that can be created by anyone that does not degrade over time and is unaffected by the elements. There will be other permanent solutions for storage, metadata, and access if we look for them.

Let's decide to move towards permanence and find solutions that last more than a few years and do not need constant care.

Digital Preservation Matters

PREMIS Data Dictionary for Preservation Metadata: Approved Changes for version 3.0

PREMIS Data Dictionary for Preservation Metadata: Approved Changes for version 3.0. Library of Congress.  November 24, 2014.
The following changes were approved for PREMIS version 3.0:

• Make Intellectual Entity another category of Object. 
• Define preservationLevelType to indicate the type of preservation functions expected to be applied to the object for the given Preservation level.
• AgentVersion was added to record the version of software Agents
• The data model was changed so that Environments can be described and preserved reusing the Object entity
• Physical Objects can be described as representations and be related to digital objects.
• A value of unknown will be added to compositionLevel and format if the information is not available.

Digital Preservation Matters.

Data Archives and Digital Preservation

Data Archives and Digital Preservation. Council of European Social Science Data Archives. June 1, 2015.
Data Archives and Digital Preservation Data archives play a central role in research. Data is considered “the new gold”. There is increasing pressure on researchers to manage, archive, and share their datadata archives. It is important to securely store research data, and to allow researchers to reuse data in their own analyses or teaching.

Archives are much more than just a storage facility; they actively curate and preserve research data. They must have suitable strategies, policies, and procedures to maintain the usability, understandability and authenticity of the data. There are also numerous requirements from users, data producers, and funders. In the social science research data preservation and sharing, archives have the added responsibility of protecting the human subjects of the research.

The CESSDA site has many resources. Some of these are:

• What is digital preservation 
• OAIS 
• Data appraisal and ingest 
• Documentation and metadata 
• Access and reuse 
• Trusted digital repositories: audit and certification.

Digital Preservation Matters.

Libraries could outlast the internet, head of British Library says

Libraries could outlast the internet, head of BritishLibrary says. Hannah Furness. The Telegraph. 25 May 2015.
Roly Keating, director of the British Library, said he was shocked at how many "smart people" still questioned whether libraries were still viable in the modern age.

Stop worrying about whether libraries will survive the digital age, that they could outlast the internet.
Libraries have countless values worth defending, including trust, that pre-dated the internet. Libraries have a vital place in communities and could prove the most "powerful and resiliant network yet". "And what we collectively believe libraries are and are for will determine what form they survive in."

 "But the time frame we think on, centuries back and centuries into the future, allows us to think about trust in its highest sense, and authentication and provenance of information, and digital information in particular.

"Those are hard-won privileges and values and they're worth defending."

"And in some ways they are the most powerful and resiliant network of all, if we continue to believe in them."

"With all our fascination of and love for the internet in the age of data, these values and the values and idea of the library predated the internet and if we get it right may yet outlast it.

Digital Preservation Matters.

Release of Archivematica 1.4 and Storage Service 0.7

Release of Archivematica 1.4 and Storage Service 0.7. Artefactual Website. May 27, 2015.
This version includes a number of bug fixes, new and updated tools plus new features. Some of the  highlights are:

• CONTENTdm integration for hosted or non-hosted environments
• DSpace integration, including capture of parent-child relationships
• Siegfried file identification tool
• Islandora/Archidora plugin
• Recovery of a backed up AIP

Digital Preservation Matters.

File identification tools

File identification tools. Gary McGath. File Formats Blog. May 26, 2015.
A series of posts on software for identifying file formats. Identifying a file means at least identifying its type, and possibly the subtype and determining if the file really is the type it claims to be. There are a variety of ways to identify the the file, and there are problems associated with these ways. Some files can belong to more than one type. Knowing the version and subtype of a file can be important. And knowing the subtypes can indicate whether or not a file is suitable for archiving.

In Linux or Unix the file command can be used to check the file, its identifiers, character encoding, and possibly the language of a text file.  The basic syntax is file --mime filename.

Digital Preservation Matters.

The race to preserve disappearing data

The race to preserve disappearing data. Bina Venkataraman. Boston Globe. May 17, 2015.
"American feature films made before 1950 faced about 50/50 odds of surviving into this century. Many independent and documentary films from the first half of the 20th century, and about 80 percent of the mostly silent movies made in the 1910s and 1920s have been lost." The cloud isn’t yet robust enough for long-term archival of complex datasets and gigantic master movie files. Nor can it keep up with predicted demand.

Three out of four feature movies screened in American cinemas are independent films, Yet most independent filmmakers have neither a plan nor budget to preserve their films. At risk is the archival footage of the future.  More than 60 percent of movie makers do not migrate their digital files to new formats. Rather than dealing with this, people want to pass the responsibility to others. This problem is not unique to the film industry but includes all areas: photos, music, documents, and including scientific research data.A crucial part of science is that researchers must be able to reproduce findings or correct them over time by re-evaluating the original data. Some of the datasets require records that span decades or longer.

A 2013 study of Supreme Court decisions  found that Nearly half of all Supreme Court decisions up to that date and more than 70 percent of law journals from 1999 to 2012 referred to Web pages that no longer existed. While the Library of Congress can preserve digital films if filmmakers share their unencrypted files, less than a dozen filmmakers and studios have done so, and the library has yet to preserve a single born-digital feature-length film. Some groups however are working on this, and digital technology can offer more automated ways to save the digital materials.

Digital Preservation Matters.

Helping Members of the Community Manage Their Digital Lives: Developing a Personal Digital Archiving Workshop

Helping Members of the Community Manage Their Digital Lives: Developing a Personal Digital Archiving Workshop. Nathan Brown. D-Lib Magazine. May 2015.

It is estimated that  93 percent of all new information is born digital. Because of the amount of digital material and the ease with which it can be deleted or lost, digital materials are very much at risk. Because libraries and librarians have the means and knowledge to care for digital materials, the responsibility to educate library users about personal digital archiving should fall on library shoulders. One way this can be done is by hosting a personal digital archiving workshop. Some questions to ask in creating such a workshop would be:

• Who is the audience?
• What kinds of digital materials do people have? 
• how do people store their digital belongings now? 
• what motivates them to maintain personal digital information?  
• What resources are available?

Some discussion points for the workshop:

1. Identify what you have and where it is located.
2. Decide what you want to keep
3. Organize what you have and how it can be identified
4. Save the digital materials and manage what you have saved

 It may also be worthwhile in discussing digital estate planning and designating a "digital executor" which should be identified in a legal will.

Digital Preservation Matters.

Saving the digital record

Saving the digital record. Kate Kondayen. Harvard Gazette. May 8, 2015.
Ensuring that digital content lives on past its initial platform is one of the most pressing issues in preservation science. This is an increasingly urgent task. Collections are increasingly coming to libraries with digital material that is already on the brink of decline. Digital degradation doesn’t follow a steady curve like physical items; it may happen suddenly and no one is sure just when that point is.

To retrieve content from an obsolete format, three components are needed: hardware, software, and a skilled technician. Many are turning to digital forensic tools. One open-source tool is XENA (Xml Electronic Normalising for Archives) which recognizes hundreds of old and unusual file formats and migrates them to current standard formats. "Even when the content is retrieved, the original media may need to be retained. Advancements now allow retrieval of content on formats that previously were written off as lost causes."

Digital Preservation Matters.

Learning from failure: The case of the disappearing Web site

Learning from failure: The case of the disappearing Web site. Francine Barone, David Zeitlyn, Viktor Mayer-Schönberger. First Monday. 4 May 2015.

This paper presents the findings of the Gone Dark Project in 2014 that looks at web sites that disappear and are no longer accessible online. A study in 2012 showed that historically significant social media content decays at a rate of 11 percent within one year, and nearly 30 percent in two years (a rate of .02 percent of resources lost per day). That is compounded by link rot, which a study of academic references showed that over 70 percent of URLs in academic journals and 50 percent found in U.S. Supreme Court opinions have broken or no longer link to the original citation information. The principal concern of the study was with sites which contain substantial or significant content. To understand the the wider landscape, the findings were categorized:

Main types of sites:

1. Scientific and Academic: Databases, research tools and repositories ranging from the natural to social sciences and humanities. Losses of this type are commonly the result of the end of funding or institutional neglect
2. Political: Personal homepages of politicians, campaign pages, political speeches and/or repositories of once-public government files.
3. Historical and Cultural: Includes collated collections of historical documents, genealogies or research portals, as well as more professionally run film, video or music archives.
4. Specialized project pages or information aggregation sites
5. Social media:

Main reasons for sites disappearing:

1. Neglect: Intentional or unintentional neglect is probably the most common reason that a site disappears
2. Technical: Technical issues are usually bundled with some form of neglect or insufficient financial resources.
3. Financial: Cost of site maintenance, staff costs, etc.
4. Natural disaster: Computer hardware is susceptible to fires, floods, rioting and neglect (just as are paper files). 
5. High-risk situations: Tumultuous political climates are a nightmare for data loss.  
6. Competition between top Web companies leads to popular services that are abandoned, shut down or absorbed into a larger platform.

 Some sites are going dark across the Web without being archived. "Restoring a single site is a challenging enough task, but when a collection of related Web sites goes dark, prevention strategies are much more difficult to specify (and quantify) as losses can potentially include an entire digital ecosystem of information". Selective archiving is best undertaken by those with firsthand knowledge of essential site. Foresight and intuition for Web preservation is not always coupled with institutional or financial stability. Solutions to the problems of sites going dark will require more awareness from all parties involved. Keeping these disappearing resources requires working with content owners to find permanent homes for at-risk data.

Digital Preservation Matters.

Why You Really Need a Digital Asset Management Workflow

Why You Really Need a Digital Asset Management Workflow. Diane Haddad. Blog. May 11, 2015.
Importance of managing your growing digital image collection. Set up a system that fits your needs and accomplishes the basic steps of Digital Photo Management. Digital Asset Management Workflow:
1. Capture images
2. Import images to a single working area
3. Rename image files from the device-generated names to follow standard naming convention
4. Backup images
5. Add appropriate metadata
6. Archive content in a permanent, off-site location
Follow the established workflow. The goal is long-term archiving and access.

Digital Preservation Matters.

Are You Doing Enough to Prevent Link Rot?

Are You Doing Enough to Prevent Link Rot?  Ernie Smith.  Associations Now. May 12, 2015.
Internet web links disappear and make web access and archiving difficult.  There are tools that can help, such as perma.cc, but what are owners of content doing to keep it in place? Research and academic institutions should be mindful of the technical needs of their underlying content, both old and new, as well as the effects that a redesign can have on a URL structure.  Content preservation requires resources, time commitment, and prioritization. However, "the sooner you create a structure for your information that is flexible and lasting, the easier these changes will be to make down the road." Some tips:

• Create a well-organized permalink structure that cites the basic subject matter and is self-explanatory
• Ensure that old links go to the right place.
• Make old content web-friendly by converting it to a format that is easier to manage on the internet, such as converting documents to html or pdf formats.

Digital Preservation Matters.

What Do We Mean by ‘Preserving Digital Information’? Towards Sound Conceptual Foundations for Digital Stewardship

What Do We Mean by ‘Preserving Digital Information’? Towards Sound Conceptual Foundations for Digital Stewardship. Simone Sacchi. Dissertation University of Illinois at Urbana-Champaign.2015. [PDF]
Preserving digital information is a fundamental concept in digital and data stewardship. This dissertation explains what successfully ‘preserving information’ really is, and provides a framework for understanding when and why failures might happen and how to avoid them. The lack of a formal analysis of digital preservation is problematic. Some notes and quotes from the dissertation:

• At a high level of generality, bit preservation means enabling the possibility for the same (set of ) bit sequence(s) to be discriminated at different points in time, and, potentially, across changes in the underlying storage technology." 
• Bit level preservation is a mean, not the goal, in digital stewardship. 
• As suggested by the OAIS definition of digital preservation, successful digital preservation is about “maintaining” or “preserving” information.
• Preserving information appears to be a metaphorical expression where a complex set of requirements needs to be satisfied in order for an agent to be presented with intended information
• The best contemporary theories of digital preservation do not focus on the preservation of any sort of object, but rather on preserving access.
• it is impossible to preserve a digital document as a physical object. One can only
  preserve the ability to reproduce the document.
• "You cannot prove that you have preserved the object until you have re–created it in some form that is appropriate for human use or for computer system applications.”
• “digital records are not stable artefacts”; they last only when certain circumstances are met
• Bit preservation is only the first required step for successful digital stewardship. Interpreting the bits such that an intended digital material obtains through appropriate performances is essential as well.
• Successful digital preservation of information can be conceived as sustained and reliable communication mediated by digital technology and agents involved in the communication process.

Digital Preservation Matters.

Binder

Binder. Artefactual Systems and the Museum of Modern Art. Github. May 2015.
Binder is an open source digital repository management application created to meet the digital preservation requirements of museum collections.  Binder aims to facilitate digital collections care, management, and preservation for time-based media and born-digital artworks and is built from integrating functionality of the Archivematica and AtoM projects. The vision is that Binder will help cultural institutions meet the challenges of long-term preservation by providing access to metadata need to inform digital preservation policy and practice.

Currently, Binder is not ready for use in a production environment, and still requires further development for the code to function in a development environment.

Digital Preservation Matters.

Robert Darnton closes the book

Robert Darnton closes the book. Corydon Ireland. Harvard Gazette. May 11, 2015.
Article about his retirement. Notes:

• He and others discussed how to harness the Internet to create a digital library that would “get our cultural heritage available to everyone” for free, leading to the DPLA
• The goal of the free digital library “was a dream of philosophers of the Enlightenment. We can do what Jefferson only dreamed of. We have the Internet, and he only had the printing press.”
• Of digital and print: "Both are complementary means of knowledge dispersal and both are thriving."
• For libraries to prosper requires advancing on two fronts, analog and digital. “We must acquire everything important in all fields of scholarship" along with “electronic outputs of all kinds, partly in cooperation with other libraries.”
• The future of libraries will require “being connected, and cooperating on a very large scale” regarding acquisition, preservation, and storage.
• The library still pumps intellectual energy into every corner of campus.

Digital Preservation Matters.

Dataliths vs. the digital dark age

Dataliths vs. the digital dark age. Gary McGath. File Formats Blog. May 4, 2015.
Digital technology has allowed us to store more information at less cost than ever before, but in return this information is very fragile in the long term. The chances that your computer’s disk will be readable in a hundred years are poor. Information needs to be stored in a form that can survive long periods of neglect. We need dataliths; this strategy requires "a storage medium which is highly durable and relatively simple to read. It doesn’t have to push the highest edges of storage density. It should be the modern equivalent of the stone tablet, a datalith."

There are devices which tend in this direction. Millenniata, quartz glass data storage, and others. "Hopefully datalith writers will be available before too long, and after a few years they won’t be outrageously expensive. The records they create will be an important part of the long-term preservation of knowledge."


[M-Discs really are the only solution along these lines at present. They are long lived, inexpensive, easy to create and read, and created according to standards. -cle]

Digital Preservation Matters.

A Digital Preservation Environment Maturity Matrix for NSLA Libraries

A Digital Preservation Environment Maturity Matrix for NSLA Libraries. Sarah Slade, David Pearson, Libor Coufal. iPres Proceeduings. October 2014.
The National and State Libraries of Australasia (NSLA) established a Digital Preservation Group to understand the state of digital preservation in the various libraries and to determine the core requirements for managing the preservation of digital collections. They listed and described the functional components of an ideal digital preservation environment and created a matrix of the current stage of development against each component for each NSLA library. Related projects by others include the National Digital Stewardship Alliance and BenchmarkDP.

NSLA Digital Preservation Environment Maturity Matrix

• Underlying Assumptions
• actively collecting digital material
• committed to preserving its digital materials for the long term.
• staff (or vendor) dedicated to the project
• intends to comply with OAIS

• Functional Component

An ideal digital preservation environment should contain a mix of policies, processes and resources (including staff and technologies). The OAIS model calls for organizations to:

• Negotiate for and accept information from information producers.
• Obtain sufficient control of the information for long-term preservation.
• Determine the designated user community.
• Ensure the information is independently understandable to the designated community without the need of special resources.
• Follow documented preservation policies and procedures.
• Make the information available to the designated community.

Instead of just listing the functions, they created a set of questions about the OAIS functionality in order to help responders describe their organization's level of digital preservation maturity, and included the OAIS functions:

• Pre-ingest Activities
• Ingest
• Archival Storage
• Data Management
• Administration
• Digital Preservation Planning
• Access

• Maturity Model

The Group modified the Capability Maturity Model, using 5 levels:

1. Initial. Processes are usually ad hoc. Achievement depends on the competence of the people in the organization and not on the use of proven processes. Products and services usually exceed budget and schedule
2.  Repeatable. Basic digital preservation processes are established.  Digital preservation achievements are repeatable, though not all activities.
3. Defined. Digital preservation activities are performed and managed according to documented plans. Processes for digital preservation are established and improved over time. 
4. Managed. Management can effectively control the digital preservation effort, using precise measurements.  Quantitative quality goal for digital preservation processes
5. Optimising. Organisation focuses on continually improving process performance. The effects of deployed digital preservation process improvements are measured and evaluated against the quantitative process-improvement objectives.   

"NSLA has identified digital preservation as an area of priority. The importance of this area to NSLA libraries is reflected in the creation of the Digital Preservation Group and its support of the Group’s work to date. The results from the Digital Preservation Environment Maturity Matrix reveal that NSLA libraries are on the right path but have some way to go before digital preservation processes are mature, sustainable and fit for purpose. Collaboration on policies, products and infrastructure will continue to address these needs."

Digital Preservation Matters.

Lessons learned in developing digital preservation tools the right way (and the wrong way).

Lessons learned in developing digital preservation tools the right way (and the wrong way). Paul Wheatley.  iPRES 2014 - Proceedings. October 2014.
There have been difficulties with creating digital preservation tools in the past, including poor technology choices, half measure in adopting open source applications, limited project funding, gaps in capabilities, poor support, and others. The wish list of tools from several years ago hasn't changed much. "Why can't we have digital preservation tools that just work?" But there have been new approaches which focus more on practical applications, sharing ideas with others and engaging with existing projects.

Lessons learned  in developing digital preservation tools:

• Be agile in the development. Develop, demo, and get feedback from users. Get crude results first, then perfect and polish later.
• Re-use, don't re-invent. There are often tools outside of the preservation community.Try existing solutions first.
• Keep it small and simple. Modularize in the face of growing requirements. Make it so the tool can be integrated into others' workflows.
• Make it easy to use, build on, re-purpose, and maintain. Share the source and package for easy install.
• Share results, knowledge; learn from others.Let others know what you are doing.

The poster includes:

• Engage with the community
• Build on existing work
• Design for longevity
• Ally with a more long lived organization as a custodian

Digital Preservation Matters.

Digital Preservation: Importance of Digitization

Digital Preservation: Importance of Digitization. Heisnam Haridas Mangangcha, Thanga Khunjem Leikai. Manipur Times. May 6, 2015.
Preservation is an umbrella term being applied to a wide variety of collection management responsibilities intended to preserve collections of print and non-print materials for future generations. The preservation of materials ensures that people's thoughts will be accessible to future generations. Digital preservation is the method of keeping digital materials ‘alive’ so that they remain usable as technological advances render hardware and software obsolete.

Digital Preservation Matters.

Digital Curation Tools & Techniques

Digital Curation Tools & Techniques. Nancy McGovern.  Webinar: Metropolitan New York Library Council. April 2015.
This is a recording of Nancy's excellent webinar and it goes along with the management tools website.  As organizations build sustainable digital preservation programs, they work toward being a trusted digital repository by following emerging digital preservation standards. There are now a number of tools to help with technical issues and now some management tools to help with organizational issues. The Digital Preservation Management workshop launched a set of management tools and techniques.

This webinar looks at using a set of digital curation and preservation principles to describe a  framework for developing a digital preservation program. The organizational aspects of digital preservation is more challenging than the technological issues. This helps people to work with tools and techniques, develop good practices, emphasize the organizational aspects and apply the 5 stages:

1. Acknowledge: understanding that digital preservation is a local concern
2. Act: initiating digital preservation projects
3. Consolidate: segueing from project to basic program
4. Institutionalize: rationalizing local efforts to establish a comprehensive program
5. Externalize: embracing inter-institutional collaboration and dependency

Each tool and technique is built on the DPM model, the three-legged stool (Organizational Leg, Technological Leg, and Resources Framework) with the five stages of development for a sustainable digital preservation program. There is a tool or technique to help organizations in each of these areas:

1. Principles: Adopt standards-based principles 
2. Policy: Develop a high-level policy framework 
3. Scope: Complete a digital content review to define program scope
4. Workflow: Document workflows to improve and automate 
5. Preparedness: Extend disaster preparedness to include digital
6. Self-assessment: Engage in self-assessment to gauge progress

Digital Preservation Matters.

Top 50 file formats in the KB e-Depot

Top 50 file formats in the KB e-Depot. Johan van der Knijff. KB Research. April 29, 2015.
Koninklijke Bibliotheek’s digital repository system doesn’t include any tools for automated file format identification yet, but an analysis of the “top 50” most prevalent file extensions shows:

• The total number of of unique extensions is no less than 1163
• The most prevalent extension is .gif
• A long tail of extensions exists (less than 10 file objects), which account for over half of all unique extensions.

The top 10 extensions are:

1. gif
2. xml
3. jpg
4. sml
5. pdf
6. raw
7. tif
8. oa3
9. doc
10. htm

They grouped all extensions into 12 format categories and checked that the PCs in the reading rooms could render the objects. The conclusion was that most formats in the “Top 50” were sufficiently accessible, though there were some improvements needed.

Digital Preservation Matters.

Quattro Pro for DOS: an obsolete format at last?

Quattro Pro for DOS: an obsolete format at last? Johan van der Knijff. Johan's Blog. October 29, 2014.
Account of researcher at the KB looking at old Quattro Pro for DOS spreadsheet and trying to access them using modern-day software. The Quattro Pro software was released in 1988. Support for Quattro Pro spreadsheets was removed altogether from more recent versions of MS Excel. LibreOffice / OpenOffice support older versions, but not the older Quattro Pro for DOS. Apart from Quattro Pro, which seemed to have some difficulties with the spreadsheets, modern spreadsheet programs offer no support for Quattro Pro for DOS spreadsheets. "The most recent version of Quattro Pro still reads both DOS era formats, although there are some problems. Some of these are formatting-related."

There were some additional tests with the Quattro Pro files. In addition, it appears that Lotus 1-2-3 spreadsheets may also be more problematic than I initially thought.

Digital Preservation Matters.

Preparing the Workforce for Digital Curation

Preparing the Workforce for Digital Curation. The National Academies Press. 2015.  
This 105 page report focuses on the need for digital curation education and training in order to provide meaningful use of digital information, now and in the future.  [PDF version] This study defines digital curation as: “The active management and enhancement of digital information assets for current and future use.” Digital curation is more than preserving the digital information in secure storage because curation may add value to digital information and increase its utility.

Digital curation is similar to traditional curation. "Regardless of whether a collection is physical or digital, a curator must appraise its value and relevance to the community of potential users; determine the need for preservation; document provenance and authenticity; describe, register, and catalog its content; arrange for long-term storage and preservation; and provide a means for access and use." But it also has many new challenges: the quantities of material to be curated, the need for active and ongoing management, continually changing uses and technology, and the diversity of organizational contexts in which curation occurs. It is more than simply collecting and storing data and information. Active management denotes planned, systematic, coordinated, purposeful, and directed actions that make digital information fit for a purpose. And to ensure that digital information will remain discoverable, accessible, and useable for as long as users have a need and a right to use it.

A new pattern of data usage puts a greater emphasis on the standardization of digital curation practices so that the data can be shared more easily.  Archiving digital data requires a more active management approach, and a more collaborative partnership between producers, archivists and users.

The Loss of Cultural Heritage Through Deterioration of Records and Technological Change: Sound recordings are a striking example of cultural heritage data at high risk of loss. These include music, oral histories, and radio broadcasts preserved in a wide variety of formats and media.

Some benefits of digital curation include:

• Increased collaboration and cost sharing;
• Greater use of data in teaching and research training;
• New opportunities and uses for data, including data mining;
• Creation of a more complete record of research;
• Creation of new areas of research, new industries, or new support services.

Some principal conclusions:

1. Significant opportunities exist to embed digital curation deeply into an organization’s practices to reduce costs and increase benefits. Digital curation will be increasingly in demand across many sectors of society.
2. Digital curation can be advanced by various organizations that can serve as leaders, models, and sources of good curation practices, and build trust by preserving assets.
3. Some barriers to digital curation include: lack of sharing of resources and insufficient resources.
4. There is a need to identify, segregate, and measure the costs of curation tasks in scientific research and business processes.
5. Standards and existing practices vary greatly, which can lead to a lack of coordination across different sectors. This in turn can lead to limited adoption of consistent standards for digital curation and fragmented dissemination of good practices.
6. Automation of at least some digital curation tasks is desirable
7. The knowledge and skills required of those engaged in digital curation are dynamic and highly interdisciplinary.

Some recommendations include:

1. Research communities, educational institutions, and others should work together to develop and adopt digital curation standards and good practices.
2. Work to identify and predict the costs associated with digital curation.
3. Organizations should identify, explain, and measure the benefits derived from digital curation

Digital Preservation Matters.

A Biological Perspective on Digital Preservation

A Biological Perspective on Digital Preservation. Michael J. Pocklington, et al. iPres Conference Proceedings. October, 2014.
Successful preservation of digital objects requires a solid theoretical framework, which treats the objects as containers of information, exactly as in the genomes of organisms.  This looks at the similarities of biological and digital ecosystems. In both cases, functional information is identifiable in principle by the consequence of actions. Interaction maps are dependency networks objects withing their environment. The significant environment information in the digital ecosystem relates to the object, resource usage, software dependencies, and other digital objects, which all related to the usability and survivability of the objects. The poster looks at an application of the theoretical background and includes first results from a case-study on a software-based art preservation scenario.

Digital Preservation Matters.

Digitization and the Preservation of Knowledge

Digitization and the Preservation of Knowledge. The Media Preservation Initiative at Indiana University Bloomington. October 10. 2013.
Indiana University President Michael McRobbie, in announcing the Media Digitization and Preservation Initiative, said in an address: “For over 25 centuries, the great universities of the world have always had three fundamental missions:

• the creation of knowledge (that is, research and innovation),
• the dissemination of knowledge (that is, education and learning), and
• the preservation of knowledge.

We tend, these days, to mainly associate the first two of these missions with a university."

The advent of the digital age is giving importance of the third mission "the preservation of knowledge". Previously, the preservation of knowledge was the almost exclusive mission of libraries and museums. Digital material is vital to fully realizing the promise of online education. The digital collections are a large investment over many years. The digital collections will continue to grow, as will the scholarly dialog concerning them, which "defines the character, values and heritage of an institution." Preserving the collections in perpetuity fully maximizes the value of all these collections to the university and the community. The transformation of the third mission of universities from the physical to the virtual world of digitization is both essential and irreversible.

Also announced was an initiative would draw all digitization efforts together into a true university-wide strategy. The goal of this Digitization Master Plan is to digitize and store in some form all the existing collections of lasting importance to research and scholarship, and to ensure the preservation of all new research and scholarship that is born digital. This will support research, education and the preservation of knowledge.

Digital Preservation Matters.