Some Assembly Required – Micro-services and Digital Preservation

Some Assembly Required – Micro-services and Digital Preservation. Danielle Spalenka. POWRR Blog. March 22, 2016.   
     Very informative article about how micro services and tools can benefit libraries of all sizes and financial abilities. Many struggle with implementing a digital preservation infrastructure.  University of California created a set of free-standing but inter-operable applications that performed a single or limited number of tasks in the larger curation and preservation process, which they described as a micro-services approach.

This approach, which did not require the installation of a single, long-lived application, can help medium-sized and smaller institutions to identify and achieve digital preservation goals. The set of twelve independent but compatible micro-services performed preservation functions such as identity, storage, fixity, replication, inventory, ingest, index, search, transformation, notification and annotation. These simple utilities would pose fewer challenges in their development, deployment, maintenance and enhancement than a large, integrated system. The strategic combination of individual services could produce “the complex global function needed for effective curation” at large institutions. The Digital POWRR (Preserving Digital Resources with Restricted Resources) Project team begin a study of the problems and possible solutions for preserving digital objects.

Some understood that digital curation and preservation was an either/or issue: either an institution had implemented a digital preservation system or it had not. In reality, the preservation activities are "an ongoing, iterative set of actions, reactions, workflows, and policies." This means institutions can begin taking small steps rather than waiting to devise an ideal solution. The NDSA Levels of Digital Preservation provides a yardstick to measure progress toward a digital curation and preservation capacity. Two fundamental understandings at the heart of a micro-services approach are:

1. digital curation and preservation is an uncertain process in which continuous, rapid technological change often renders monolithic, integrated applications cumbersome and outdated;
2. simple tools focused on a specific aspect or aspects of the process can prove more helpful.

Micro-services tools can only be effective if users understand what roles they play in the larger digital preservation process and the path they take through the NDSA Levels. The COPTR (Community Owned Digital Preservation Tool Registry) web site provides information about many helpful tools and services that can provide incremental value.  Micro-services tools can also help those adopting more robust tools.

The use of individual tools performing discrete functions can help those starting preservation activities. The Digital POWRR Project has described the stages of a digital curation and preservation workflow and associated activities. Tools performing these functions are available at the COPTR web site.

Digital Preservation Matters.

Exploring appraisal, quality assurance and risk assessment in the data continuum

Exploring appraisal, quality assurance and risk assessment in the data continuum.   Linda Ligios. Pericles blog. 8 March 2016.
     PERICLES presented a workshop on "appraisal, quality assurance and risk assessment in relation to the lives of complex digital objects."   It introduced the key concepts of :

• model-driven preservation in a continually evolving environment
• appraisal processes that lend themselves to being automated,
• development plans for tools on appraisal, risk assessment and quality assurance 

Three main dimensions

1. Risk – probability of an entity being non-usable
2. Proximity – time frame in which we consider risk/impact
3. Impact – potential loss of functionality and cost of mitigating actions

"Policies should always reflect the vision of the institution and therefore contain principles that are more aspirational in nature."  The  PERICLES model-driven preservation approach:



Related topics:

• PERICLES: towards model-driven preservation. Poster by Jean-Yves Vion-Dury.
• PERICLES Environment Extraction Project 

Digital Preservation Matters.

Translating theory to practice : defining digital preservation planning in museums

Translating theory to practice : defining digital preservation planning in museum. Emma Palakika James. Thesis, San Francisco State University. January 2016. [PDF, 292 pp.]
    A very interesting thesis looking at digital preservation as an emerging activity in museums today. Some of the chapters discuss Threats to Digital Objects; What is Digital Preservation? (including the basics, OAIS, Trusted Digital Repository, methods; case studies, and Digital Preservation Policy.  Some notes of interest:

•  The development of technology as a tool for work, research, information capture, and artistic expression, as well as the increasing percentage of important cultural materials created only in digital form, argues that museums must begin to focus on digital preservation.
• Four key themes are discussed, including 
• defining digital preservation,
• integration of digital preservation technology, 
• collaboration, and 
• policy development
• currently, digital preservation remains a new, and not-broadly practiced activity in museums. 
• The practice of digital preservation will become increasingly important to the museum field, and should be considered with the same responsibility and effort as traditional museum collection management. 
• If museums are going to continue their role as well-equipped stewards for the cultural heritage of today and of our future, then digital preservation will need to be adopted within the broader scope of museum work.
• digital preservation in a museum context must be viewed and implemented from a collections management perspective.
• In some form or another, eventually all museums will adopt digital technology into their institutional assets, museum archives, and museum collections, all of which will continually be expected to be cared for and preserved just as long as any analog collections.
• While digitization was the very beginning of increased public access to collections, digital preservation is simply the flip side of ensuring ongoing access — providing consistent entry to information that is already manifested in digital form. 
• If access to collections is becoming a mainstreamed part of the Museum’s responsibility, the ongoing access to born-digital institutional assets is also certainly worthy of consideration. 
• Storage has been the default long term preservation strategy used by museums for traditional collections, but it is the shortest-term solution for new media
• American Institute for Conservation of Historic and Artistic Works: “every institution has a responsibility to safeguard the collections that are entrusted to it. That responsibility includes incorporating preservation and conservation awareness into all facets of the institution’s activities so as to ensure the long-term preservation of its collections”
• active digital preservation tactics should be accessible, manageable, and realistic solutions.
• Collaboration is necessary to ensure preservation and especially if the museum field wants to improve itsstewardship of digital materials
• collaboration between libraries, archives, and museums will be a critical factor for whether the greater museum field can achieve digital preservation to the level of a ‘Trusted Digital Repository,’ which arguably, is the ideal level of preservation for medium to long-term stewardship. 
• Together, LAMs can ensure that ‘bom-digital’ documents and artifacts become integrated into the cultural record through various levels of digital preservation activity that will help to keep them accessible, and to become a permanent part of the cultural memory of future generations.
• most museum collection management policies do not address the issues of digital preservation or digital stewardship.
• the need to draft and implement a digital preservation policy is of equal importance to that of collection management policy for a museum.
• a full-formed policy is a way for the staff, and hopefully eventually upper management, to organize the overall mission, goals, scope, staff roles, and basic procedures. This may help better define how the staff can tackle digital preservation, making it a less intimidating process and to also document its official initiation 
• the technology the three case studies falls into three categories: digital asset management systems, OAIS compliant software, and storage media.
• is a digital asset is not a digital preservation system in of itself, because a DAMS does not usually follow the specific recommendations for metadata, fixity checks, and formats that are put forth by Trusted Digital Repository model, OAIS and other standards
• Digital preservation systems ultimately are a set of processes, protocols, and policies that are most often mediated with some technological aspect to aid in creating information packages suitable for long-term storage
• Digital preservation is not just a technology problem, but it is a management issue. 
• Policy is a tangible method for institutions to outline the management support of their preservation activities

The thesis includes five conclusions concerning the state of digital preservation in museums:

1. preservation is possible; 
2. standards, guidelines, and best practices are already available, but use wisely; 
3. embrace new practices in policy;
4. collaboration will be key for success; and 
5. embrace change and act now.

“To create a collection, to inherit one, or to be given oversight of a collection, is also to create, inherit, or accept a great responsibility”

Digital Preservation Matters.

Caring for file formats

Caring for file formats. Ange Albertini. Presentation at Troopers 2016. March 17, 2016. [PDF]
     The risk to preserving digital objects is very high. The "attack surface with file formats is too big". The specifications of formats are a nice guide, but they don't represent reality; they are useless for managing the formats. "We can’t deprecate formats because we can’t preserve and we can’t define how they really work."
The formats need good documentation to show the landscape and "to express the reality of file format".  Once they are better understood, then "we can preserve and deprecate older format, which reduces attack surface". Then people can focus on making the present formats more secure.

What is a file format? A computer dialect to communicate between communities; file formats are community connectors. People don't care about the format itself, they care about the characteristics and how easy it is to use. We don't need new formats, since reality will diverge from the specs anyway. The need is for up to date, traceable specs. Formats are constantly being updated with new features added. That doesn't solve the problem.  Specs should reflect reality and be "updated, enforced, realistic, freely available". Deprecation is a natural cycle, but are afraid to deprecate because "no file format is fully preserved". Formats should be open and the specs kept up to date. But it won’t happen until "we experience a great disaster".

Digital Preservation Matters.

National Archives permits us to learn from mistakes

National Archives permits us to learn from mistakes. Peter Charleton, Supreme Court judge. The Irish Times. Feb 8, 2016.
      For the National Archives in Ireland, 1922 was a disaster. A direct hit from artillery destroyed centuries of records. The census records from 1821 through to 1891 were almost completely destroyed. Since then, the National Archives has tried to supplement its damaged holdings, but what has been lost is gone forever. With many places moving from paper to digital records history is on the point of repeating itself. The traditional policy of printing files to preserve a digital record no longer works. Files may be on several computers in several iterations; they may have "elements in office systems, email, even text messages or a tweet." With digital, there is a lot of data, which brings a challenge of what to preserve.  But not every record needs preservation.

To ensure records the preservation of long-term records, the records should be transferred to the National Archives. Permanent records need to be identified early and treated appropriately.  Creation of a digital archive will greatly reduce the volume of records that government departments store. "Millions are spent by departments on off-site storage and back-ups of network drives. By investing in a digital archive, departments will be able to transfer emails, business files, digital images and other electronic records to the National Archives. An efficient approach to records management based on legal obligation can target policy effectively."  Money should be directed to the National Archives for developing an efficient system so there are sufficient resources to capture, manage and preserve our digital heritage. This institution, the precious repository of this nation, deserves to be supported in ensuring Ireland continues to have a history.

Digital Preservation Matters.

The FAIR Guiding Principles for scientific data management and stewardship

The FAIR Guiding Principles for scientific data management and stewardship. Mark D. Wilkinson, et al. Nature. 15 March 2016. [PDF]
     "There is an urgent need to improve the infrastructure supporting the reuse of scholarly data." Good data management is not a goal in itself, but a conduit leading to knowledge discovery, innovation and the reuse of the data. The current digital ecosystem prevents this, which is why the funding and publishing community is beginning to require data management and stewardship plans. "Beyond proper collection, annotation, and archival, data stewardship includes the notion of ‘long-term care’ of valuable digital assets" so they can be discovered and re-used for new investigations.

This article describes four foundational principles (FAIR) to guide data producers and publishers:

• Findability,
• assigned a globally unique and persistent identifier
• data are described with rich metadata
• metadata clearly include the identifier of the data it describes
• data are registered or indexed in a searchable resource
• Accessibility,
• data are retrievable by their identifier using a standardized communications protocol
• the protocol is open, free, and universally implementable
• the protocol allows for an authentication and authorization procedure,
• metadata are accessible, even when the data are no longer available
• Interoperability, 
• data use a formal, accessible, shared, and broadly applicable language for knowledge representation
• data use vocabularies that follow FAIR principles
• data include qualified references to other (meta)data
• Reusability
• meta(data) are richly described
• (meta)data have a clear data usage license
• (meta)data have a detailed provenance
• (meta)data meet community standards

These FAIR principles guide data publishers and stewards in evaluating their implementation choices. They are a prerequisite for proper data management and data stewardship. Achieving these goals requires working together with shared goals and principles.

Digital Preservation Matters.

New Report on Web Archiving Available

New Report on Web Archiving Available. Andrea Goethals. IIPC. 21 March 2016.
     Harvard Library recently released a report to:

• explore and document current web archiving programs
• identify common practices, needs, and expectations in the collection of web archives
• identify the provision and maintenance of web archiving infrastructure and services;
• identify the use of web archives by researchers.

The environmental scan showed 22 opportunities for future research and development, which includes:

• Dedicate full-time staff to work in web archiving to keep up on latest developments, best practices and be part of the web archiving community.
• Conduct outreach, training and professional development for existing staff who are being asked to collect web archives.
• Institutional web archiving programs should be transparent about holdings, terms of use, preservation commitment, are curatorial decisions made for each capture.
• Develop a collection development tool to show holdings information to researchers and other collecting institutions.
• Train researchers to be able to analyze big data found in web archives.
• Establish a standard for describing the curatorial decisions behind collecting web archives.
• Establish a feedback loop between researchers and the librarians/archivists.

There is also a need to "radically increase communication and collaboration" among all involved in web archiving. Much more communication and collaboration is needed between those collecting web content and researchers who would like to use it.

Digital Preservation Matters.

Do your Digital Records have an Expiration Date?

Do your Digital Records have an Expiration Date? Jon Tilbury. Information Management. March 22, 2016.
     A general article about the importance of digital preservation.  Some quotes of interest:

• As more “born digital” content is produced every day, requirements get more complex, and the need for organization-wide digital preservation strategies becomes greater.
• the consensus is that the “tipping-point” for accessing digital data is not 100 years after it is stored, but more realistically, around 10 years.
• Ten years is a more realistic time-frame to consider when planning protection for critical and unique digital information assets. 
• Building on reliable storage, digital preservation adds tools to accurately identify which formats are being used, pinpoint those at risk, and reliably recycle these into newer formats that can be read.
• For many organizations, a proactive approach to safeguarding critical long-term digital records t using digital preservation technology is fast becoming a critical part of the overall information governance lifecycle.

Digital Preservation Matters.

From digital dark age to digital enlightenment

From digital dark age to digital enlightenment.  Caroline Pegden. National Archives. 17 February 2016.
     Recent media reports have talked about the ‘digital Dark Age‘.  This is a major challenge, now and for the years to come for institutions in the archives sector, who are concerned with managing, preserving and providing access to born-digital records. This is important for the UK National Archives because some government departments will soon transfer born-digital records to The National Archives under the Public Records Act. As the National Archives has been working on how to do this, their philosophy has been ‘learning by doing’. They have reviewed what other archival institutions around the world are doing to manage digital records and have been testing the process of transfers "to design and test the new process to appraise, select, sensitivity review, transfer, preserve and give access to born-digital records." Two major challenges are:

1. extracting meaning from unstructured digital record collections in order to make appraisal and selection decisions.
2. sensitivity reviewing born-digital records at scale without having to read all the individual documents

Most government departments’ information is on unstructured shared drives; some departments had up to 190 terabytes of information in email servers.  Technology-assisted-review, a process using reviewers and a combination of computer software and tools to electronically classify records may have interesting applications for the archives sector. "Although there is no ‘silver bullet’ or completely automated solution, technology-assisted review offers ways to prioritise and reduce the information to be manually reviewed."  Two reports are available that highlight challenges and shows how technology-assisted review could help addressing some of these challenges. 

1. The digital landscape in government 2014-15: business intelligence review
2. The application of technology-assisted review to born-digital records transfer, Inquiries and beyond: research report

Digital Preservation Matters.

How many of the EOT2008 PDF files were harvested in EOT2012

How many of the EOT2008 PDF files were harvested in EOT2012.  Mark Phillips. mark e. phillips journal. February 23, 2016.
     Post aabout  the author looking at some of the data from the End of Term 2012 Web Archive snapshot at the UNT Libraries. From the EOT2008 Web archive 4,489,675 unique (by hash) PDF files were extracted and then compared recently to see how many of those nearly 4.5 million PDFs were still around in 2012 when they crawled the federal Web again as part of the EOT2012 project. The findings:

After the numbers finished running,  it looks like the following.

                       PDFs        Percentage
Found             774,375       17%
Missing        3,715,300       83%
Total            4,489,675     100%

So 83% of the PDF files that were present in 2008 are not present in the EOT2012 Archive. It is possible that the items are still available at a different URL entirely in 2012 when it was harvested again. So the URL might not be available but the content could be available at another location.

Digital Preservation Matters.

Preservation Watch

Preservation Watch. Barbara Sierman. DPC wiki. 12 February 2016.
     Preservation Watch is a well-accepted concept that was first created during the European PLANETS project by Barbara Sierman and Paul Wheatley. It goes beyond the OAIS monitoring activities and helps provide a better description of the Preservation Planning Functional Entity. The Planets Functional Model identifies 3 key preservation functions:

1. Preservation Watch, 
2. Preservation Planning and 
3. Preservation Action.

Preservation Watch monitors internal and external entities, including the repository content. It deals with both the OAIS Administration and Preservation Planning areas. The Preservation Watch has 4 sub-functions:

1. Monitor: collates preservation information from a variety of internal and external entities.
2. Risk Analysis: assessment of this information, relaying critical risks to Preservation Planning.
3. Representation Information Update: provides updates, including recording Risks and Executed Preservation Plans.
4. Testbed:  a controlled environment for studying the operation of tools and services which will inform the Preservation Planning activities.

Digital Preservation Matters.

'A' is for AtoM

'A' is for AtoM. Jenny Mitcham. Digital Archiving at the University of York. 18 March 2016.
     Jenny has been working on an implementation of Access to Memory (AtoM) for a couple of years and provides an interesting list of information about it, the A to Z* of implementing AtoM. "It turns out that deciding to adopt a system is relatively simple, working out exactly how you are going to use it is far more complex!" A few that I found that apply in most software situations:

• B is for Business as Usual: Any organisation when adopting a new and complex system like AtoM needs to think beyond initial implementation and consider how the solution can be embedded into their workflows for the longer term?
• E is for Experimenting: We discovered that data may not always import in the way you expect.
• J is for Just Start!: Reading the documentation is essential but testing and experimenting with AtoM are really the best ways of working it out.
• N is for Not Perfect: AtoM (like all complex systems) has its limitations.
• Q is for Quality: In an ideal world, all our data within AtoM would be of a high quality....but we do not live in an ideal world. Accepting that legacy data will not always meet current standards or be as accurate as we would like is key to moving forward with a system such as this.
• T is for Training: Training is not just a one off exercise.

Digital Preservation Matters.

Applying DP Standards For Assessment & Planning

Applying DP Standards For Assessment & Planning. Bertram Lyons. PASIG 2016. March, 2016.
     ISO 16363:2012. Audit & Certification of Trustworthy Digital Repositories defines recommended practices for assessing the trustworthiness of digital repositories. The document will help those who audit repositories, but also those to design or redesign their digital repository processes. Some highlights from the standard:

3.1 Governance and Organizational viability: The repository shall have a collection policy or other document that specifies the type of information it will preserve, retain, manage, and provide access to. Without the policy the collection scope is unclear and it becomes difficult to say no to out of scope content. The standard expects a policy to exist and be documented.

4.2 Ingest: Creation of AIPs: Organizations should have a description of how AIPs are constructed from SIPs. It should document all changes to the processes, as well as defining what happens to the content (such as normalization of files, etc.)

5.2 Security Risk Management: The repository should have a written disaster preparedness and recovery plan, including at least one off-site backup of all preserved information together with an off-site copy of the recovery plan. This means the organization should be prepared administratively.

The elements are scored as follows

• 0 - non-compliant or not started
• 1 - slightly compliant (needs a lot of work to do in address the requirement.
• 2 - half compliant: partially addressed but still significant work to do
• 3 - mostly compliant: mostly addressed and working on full compliance.
• 4 - fully compliant: can demonstrate the requirement is comprehensively addressed.

Elements needed:

• Documentation: records of policy, procedure, and outcomes of activities
• Policy: the definition of approaches and protocol for repository functions and procedures
• Procedures: specification of preservation and infrastructure management activities
• Software: development or configuration of preservation systems
• Infrastructure: procurement, monitoring, and management of hardware infrastructure
• Organization: organizational infrastructure including funding, staffing, and strategy
• Action Plan

Digital Preservation Matters.

Guidelines for the selection of digital heritage for long-term preservation

Guidelines for the selection of digital heritage for long-term preservation.  UNESCO/PERSIST Content Task Force. March 2016.
     Libraries, archives, and museums traditionally have the responsibility of preserving the intellectual and cultural resources produced by society but this is in jeopardy because of amount of information created every day in digital form. Digital content is doubling in size every two years.The digital content is also in danger because much of it is ephemeral; it lacks the longevity of physical objects. The challenge of keeping digital content "requires a rethinking of how heritage institutions identify significance and assess value". Institutions must be proactively identify and preserve digital heritage and information before it is lost. The role of libraries, archives, and museums are blurring in the digital age, but they still have major interests to preserve heritage.

Libraries face the challenge selecting digital content for long-term preservation. Many focus on short term use content already in their collection, rather than assessing new publications for acquisition. Archives have traditionally "relied on the passage of time between their creation and their acquisition by an archive to lend historical perspective in making selection decisions". However, the time frame for selecting content is shorter now since the rapid obsolescence of digital formats, storage media, system hardware and software systems, of opportunity of selection.  Some strategies for selecting digital content:

Acting locally 1: Strategies for collecting digital heritage.

• Comprehensive collecting to acquire all of the material produced on a given subject area, time period, or geographic region.
• Representative sampling to capture a representative picture makes selection and preservation more manageable and less resource-intensive.
• Selecting material for addition to their collections based on specific criteria, such as
• Subject/Topic.
• Creator/Provenance.
• Type/Format.
• Institutions could defer selection by capturing all the digital heritage material now and apply selection criteria later.

Acting locally 2: Developing selection criteria for a single institution

How should institutions select, identify, and prioritize digital heritage before it is lost? Evaluating and assessing digital content should be based on the principles that underlie traditional selection, but include long term perspective for use and access as defined by its mandate and users.

Decision Tree for Selection in an individual Institution

1. Identification. Identify the material to be acquired or evaluated. 
2. Legal framework. Does the institution have a legal obligation to preserve the material?
3. Application of three selection criteria to determine if content should be preserved: significance, sustainability, and availability
4. Decision. make a decision based on the three items and then document the rationale and justification for the evaluation or decision.

"The long-term preservation of digital heritage is perhaps the most daunting challenge facing heritage institutions today. Developing and implementing selection criteria and collecting policies is the first step to ensuring that vital heritage material is preserved for the benefit of current and future generations."

Appendix 1: Management of long-term digital preservation and metadata. If the digital heritage is the “content”, then the metadata provides the “context”.

"Selection of digital heritage is closely connected with issues related to long-term preservation and access. Some losses of important digital heritage may be unavoidable, but the risk can be mitigated by following best practices in digital preservation, including redundancy, active management, and metadata management."

Three key types of metadata crucial to long-term preservation:

• Structural (required for the technical capacity to read digital content)
• Descriptive (containing bibliographic, archival, or museum contextual information, which can be system-generated or created by heritage professionals, content creators, and/or users)
• Administrative (documenting the management of a digital object while in its collection).

Five basic functional requirements for digital metadata:

1. Identification: The metadata must identify each digital object uniquely and unambiguously.
2. Location: The metadata must allow each digital object to be located and retrieved.
3. Description: A description of digital object as well as data about the content and the context.
4. Readability: Metadata about the structure, format and encoding of digital objects
5. Rights management: Rights and conditions of use and restrictions must be recorded.

Digital Preservation Matters.

File identification ...let's talk about the workflows

File identification ...let's talk about the workflows. Jenny Mitcham. Digital Archiving at the University of York. 27 November 2015.
     When adding files to a digital archive, an important questions is "What file formats have we got here?" Knowing this can:

• determine the right software to open the file and view the contents 
• start the conversation with the data provider about what formats are best to use for archiving
• discuss the risks on the format and define a migration pathway for preservation and/or access

There are many tools for working with formats; each tool has strengths and weaknesses. Defining a workflow can help determine how best to use these tools, how to interact with them, or if manual steps should be taken instead. File identification tools are often incorporated into digital preservation systems that may determine the workflow in using the tools. Additional workflow questions around format tools include:

• what should happen if ingested data can't be identified?  
• should the curator/digital archivist be able to over-ride file identifications?
• what should happen if there is more than one possible identification for a file?
• is there a sustainable manual identification process if tools cannot identify a file? 
• how to contribute to file format registries such as PRONOM
• is the digital preservation system configurable enough to resolve these questions? 

Their Archivematica development work is focusing in the first instance on allowing the digital curator to see a report of the files that are not identified in order to understand the problem.

[Our Rosetta system has a format library that handles these questions, as well as a user driven Format Working Group that helps resolve questions and interacts with PRONOM if there are questions, changes or new additions. - Chris]

Digital Preservation Matters.

The Digital Preservation Network (DPN) Has Launched and Is Accepting Content

The Digital Preservation Network (DPN) Has Launched and Is Accepting Content.  Mary Molinaro. D-Lib Magazine. March/April 2016.
    Several years ago a group of academic leaders examined the risk to future scholars if the digital output from academia is not properly preserved and felt that the risk of loss was very high if nothing was done to protect against natural disasters, technological failure, or institutional failure. They pledged to create a large-scale digital preservation service that is built to last beyond the life spans of individuals, technological systems, and organizations. After three years of work, the resulting Digital Preservation Network is open and is accepting content from members. Five preservation repositories make up the DPN network. They have varying technical architectures and replicate content and perform services to safeguard the content. Content from member institutions can be added to DPN through two sites: DuraCloud Vault and the Academic Preservation Trust. The deposited content is replicated to the other nodes (Hathitrust, the Texas Preservation Node, and the Stanford Digital Repository).

DPN operates as an independent organization under the umbrella of Internet2 and is currently examining ways to open up DPN to other kinds of members.  More information is available at the DPN website.

Digital Preservation Matters.

Archives and SharePoint

Archives and SharePoint. Heather Emily Roberts. HerArchivist. March 8, 2016.
     Post that looks at "Is SharePoint (or other flexible cloud-based ERMS software) suitable for digital repositories of archives?"  Some pros and cons of using SharePoint as a digital repository:
Pros:

• Can lock documents against editing
• Tells you when documents were last accessed and by whom

Cons:

• Will not serve long-term needs of accessibility or use of records
• Will not support migration requirements of archival records
• Will not guarantee integrity of archival records during software updates
• Does not conform to OAIS model
• Archive preservation practices are not standard

The main recommendation if using SharePoint as an Electronic Record Management System is to export archival documents to an OAIS compliant system (some are listed).
[We use our harvest tool to import permanent SharePoint records into our Rosetta system - Chris.]

Related posts:

• New Sources and Storage Options For Rosetta
• OAI-PMH harvesting from SharePoint

 

Digital Preservation Matters.

Digital Preservation - Knowing where to start

Digital Preservation - Knowing where to start. Nik Stanbridge.  Cloud Computing Intelligence.
23 February 2016.
     Memory institutions face increasing demands on their collections, such as the need to manage costs better, provide access, or degradation of objects, which then require digital preservation. Some organizations already have a strategy and are digitally preserving their assets. Many though are only just starting to think about digital preservation and need to know where to start and how to implement digital preservation.

Digital preservation is the process of managing and storing digital files and associated metadata in a way that they will be accessible and usable in the future. The processes apply both objects that were originally created in digital form and to those that have been digitized. If you have a need to maintain digital objects then the first step is to define a strategy; understand what needs to be preserved and how. This includes information about the digital object. "It’s important to remember that digital preservation is as much about preserving the meaning and context of the asset as it is about preserving the asset itself."

• File format preservation is the process of maximising the accessibility of the file through its repeated migration to any number of more stable or current file formats. 
• Data archiving is the process of storing all of the resulting digital assets for the long term, using active archiving principles and processes.  
• A preservation strategy will also need to cover the people and processes you are going to use, the quality of the digital assets to preserve and the IT infrastructure and associated support.

Most strategies will need a phased approach that addresses the digital preservation factors that are important to the institution. Establishing a digital preservation strategy is the very first step. Consider carefully also how the digital objects will be stored long term.

Digital Preservation Matters.

Demystifying Digital Preservation for the Audiovisual Archiving Community

Demystifying Digital Preservation for the Audiovisual Archiving Community. Kathryn Gronsbell, Abbey Potter. The Signal. February 22, 2016.
     "The intersection of digital preservation and audiovisual archiving has reached a tipping point." Media production and use as well as the preservation strategies, including improvements in digital capture technology, adoption of file-based production workflows, digital distribution technology. storage solutions, over the past decade we have witnessed a series of transformations that fundamentally alter dominant theories and practices of moving image preservation and access.  The acceptance of digital preservation has been slower in the moving image archiving and preservation community than in other fields. Rarely are the challenges of preserving audiovisual materials discussed.  Recent proposals for audiovisual preservation include:

• Transition to a stream-based preservation model
• Digital preservation in practice (strategies)
• Discussions on how to preserve (innovation and practical engagement)
• The necessity of multi-disciplinary input for preservation
• Transitioning from a short-term digital preservation project to a long-term program (sustainability)

Preservation in the moving image community may be slower because resources must be devoted to managing the physical collections, as well as the cost and complexity of preserving analog film/video content. "The shifting focus towards digital preservation is an opportunity to dissolve the manufactured boundary between A/V and still image (or other) content and include audiovisual specialists in broader discussions of preservation and access." There is a pressing need and collective desire to address some of the questions that digital preservation raises.  "Increasing the engagement of the analog film and video world with the digital preservation community, and vice versa, will yield tremendous benefits on both sides of the divide." Digital preservation should be understood as a core competency in the A/V archiving field and become part of a wider conversation about digital preservation across disciplines.

The AMIA organization hopes to bring together those who have limited resources or haven’t started strategically thinking about digital preservation, a place where the A/V community can learn without feeling lost in a wave of information. Hopefully this will increase the visibility of the intersection between audiovisual preservation and digital preservation continue the conversation between these two fields.

Digital Preservation Matters.

How Digital Storage Is Changing the Way We Preserve History

How Digital Storage Is Changing the Way We Preserve History. Arielle Pardes. Vice. February 19, 2016
     Article starts with an account of a digital diary platform called Oh Life; after the site had been shut down, thousands of archives were deleted and years of personal history were gone.  Digital disappearance like this is a warning sign to historians of problems to come with recording and preserving our history in the digital age. Digital storage is fragile and the files can easily be lost or locked up in encryption. Digital technology might not be around tomorrow, and many of the information storage platforms are owned by private companies, which makes it harder for archival institutions to save them.  Abby Smith Rumsey tried to troubleshoot how to store digital materials in the long-term and discusses concerns and possible future solutions for our digital age.

• In the digital age, there's a lot circulating in the way of information, but none of it is kept very thoroughly. 
• Technically, we don't know how to preserve it yet. Even more than that, what do we preserve? How do we know what's valuable?

Entire digital archives can vanish if the storage platform, technology, or software disappears. Many of the websites we use are owned by private companies and individuals do not own the content. We won't know for a while if the content we have saved is valuable in the future. "The more the mind can be freed of certain types of memory tasks, the freer the mind is to engage in other activities that machines cannot do for us."

Digital Preservation Matters.

Teaching Files: Incorporating Born-Digital Materials into Instruction

Teaching Files: Incorporating Born-Digital Materials into Instruction. Dorothy Waugh. CurateGear 2016. January 14, 2016.

• Establish a sustainable model for the integration of born-digital materials into instruction programs
• Help faculty and students use born-digital collections and promote greater understanding
• Explore requirements and policies needed to use born-digital materials in the classroom and library
• Train Library staff to talk confidently  about born-digital materials and provide support for research methods using born-digital materials

Voyant Tools: Voyant Tools is a web-based text reading and analysis environment to work with texts in a variety of formats, including plain text, HTML, XML, PDF, RTF, and MS Word.

Digital Preservation Matters.

Wowza as a Streaming Media Service at the National Library of New Zealand

Wowza as a Streaming Media Service at the National Library of New Zealand. Mathachan Kulathinal. Rosetta Tech Blog. March 2, 2016.
     The National Library of New Zealand uses the Wowza Streaming Engine Server to stream derivative copies of video and audio content stored within their Rosetta preservation system. Details of our integration with Rosetta are outlined below, configuration or environment specific details used in the implementation of this model at the Library have not been included.

According to the diagrams, the Wowza Streaming Engine service is separate from the Rosetta servers; the streaming media servers have been kept separate and independent of the Rosetta services. The servers are also used as proxy servers to re-direct http traffic as required. The post includes the implementation details.

Digital Preservation Matters.

The Human Face of Digital Preservation: Organizational and Staff Challenges and Initiatives at the Bibliothèque nationale de France

The Human Face of Digital Preservation: Organizational and Staff Challenges and Initiatives at the Bibliothèque nationale de France.  Emmanuelle Bermès, Louise Fauduet.  iPres, October 2009.  [Video, full paper, slides]
     Great presentation. The National Library has been working for several years on their digital preservation efforts, with the Spar  project (Système de Préservation et d'Archivage Réparti). They are looking at the human aspects of their digital projects. The library has become digital as a whole, which was a major change. Earlier the digit library was treated differently from the rest of the library. Originally the digital side was led by experts or early adopters who were learning by doing and were organized separately from the rest of the library workflows. Digital definitely meant "different". Now the library has become digital, which means  they have regular production teams running operational projects for digital content and digital preservation.  All organizations within the library are involved in these tasks and there are formal training processes. The library shifted and part of this was a large scale shift in the scale of content digitized, as well as making digital activities closely related with traditional library skills. If you want to disseminate the digital activities through out the library you have to disseminate the tasks and the people as well. You have to take the time to train everyone and move slowly to bring all people along, or you leave people behind.

Many of the digital activities can be related to the other workflows, like ingest and acquisitions, metadata, cataloging, etc. Relate traditional librarian skills with digital skills; digital can be built on traditional library knowledge. Help integrate the digital by having common projects that people can work together on. It is important to take time to stop and look back at what has been done and talk about it. Difficult to take advantage of what has been done if you don't review. The library took time to move the conceptual OAIS model into the reality of the library organization, to decide how does it really fit, define roles and decide how the employees would interact.

The Library created a digital training curriculum and opened it up to everyone in the library to learn. "This training includes an introduction to digital libraries, digital documents, and digital preservation, and then three optional tutorials, one on metadata and protocols (including semantic web technologies), one on user oriented design (including usage studies, accessibility and usability, and the Web 2.0), and one on digitization and preservation (including rights management, preservation metadata and persistent identifiers)." They had to draw a line between those who wanted the training for their job, or those who were just curious. They started a project about organizations and human resources under digital influence to better understand the digital library and their people. Six wishes:

1. Clarify the institution’s policy and digital strategic vision
2. Define priorities 
3. Define what a digital collection is
4. Facilitate transverse workflows that span easily across organizational borders
5. Develop digital skills
6. Analyze job qualifications, revisiting job requirements and updating staff skills

Digital Preservation Matters.

Assessing Digital Preservation at the John F. Kennedy Presidential Library

Assessing Digital Preservation at the John F. Kennedy Presidential Library. Alice Sara Prael, Abbey Potter. The Signal. March 2, 2016.
      The question is with all the data in the library holdings, how to preserve the digital files over the long term? The goal of the project is to ‘develop a long-range digital preservation strategy’ to address all digital archival holdings at the Library. This challenging goal consists of three phases:

1. assess current infrastructure against community standards and make brief recommendations on how to improve digital preservation practices
2. explore potential solutions to address the recommendations made in the first phase
3. determine a single path forward based on the solutions explored and create an action plan for how to implement that solution.

Plans started by interviewing archivists and IT personnel about their processes and how they use the systems and then research the systems. Like many cultural heritage institutions, is in need of better documentation. The biggest gap in the existing documentation for digital archives is a digital preservation policy which is a record of decisions, including those that have been made but not documented or documented elsewhere. Important community standards and guidelines include

• ISO 14721: Reference Model for an Open Archival Information System (OAIS), 
• ISO 16363: Audit and Certification of Trustworthy Digital Repositories,
• the National Digital Stewardship Alliance Levels of Digital Preservation

Each of these items gives a slightly different perspective on what is required for digital preservation. With limited resources and staff time it’s important to recognize when to aim for “good enough” digital preservation, which can be defined for each institution by "the available resources, the needs of the collection, and priorities of the institution".  The NDSA Levels of Digital Preservation are not as in depth as but they are easier to understand. The intermediary levels can address digital preservation in a phased approach and also create a way for identifying strengths and weaknesses. Once the NDSR project is complete the Library will have a picture of the digital practices and a clear implementation plan for improved digital preservation.

Digital Preservation Matters.

Pursuing the "Long Tail" of Elusive Publishers

Pursuing the "Long Tail" of Elusive Publishers. Stephanie Orphan, Amy J. Kirchoff, Kate Wittenberg. Center for Research Libraries. Webinar. Wednesday, February 24, 2016.
     Pursuing the "Long Tail" is the first in a series of four CRL webinars that will examine the benefits, capabilities and possibilities of the Portico digital preservation service.

• Presentation Slides
• Supplemental Spreadsheet: Comparing Archived Title Coverage
• Youtube Presentation

The archived title coverage information shows what titles are not archived in Portico, CLOCKSS, LOCKSS, or JSTOR, compared to the total number of journals listed by CrossRef. The slide shows:

• Titles Preserved Somewhere: 28,473 (62%)
• Titles Not Preserved Anywhere: 17,388 (38%)

In the slide, there are 28 publishers that publish over 200 titles, while there are 4,650 that publish 1 or 2 titles and they have 4,039 titles that are not preserved anywhere. That will take a long time to include all those publishers. Libraries can help by:

• Providing the names of specific publishers that are a high priority.
• Providing categories of publishers or  platforms that are a high preservation priority.
• Talking to publishers about participating in a preservation service

Digital Preservation Matters.

The M Word: The Good, the Bad, the Ugly

The M Word: The Good, the Bad, the Ugly. Robert H. McDonald and Juliet L. Hardesty. Educause Review. January 11, 2016.
     Metadata is a part of all aspects of the research process. "The expectations are that metadata will be clean and understandable, secure and accessible when appropriate, and easily shareable." In reality this doesn’t happen naturally or without "concerted effort and cooperation" in all areas of policy, design, and practice. Metadata created by hand can be problematic; the academic research librarians work with metadata to "ensure good storage, maintainability, shareability, and most importantly, accessibility."

Libraries are often the right agency to serve as a neutral mediator for collaborations among researchers. New directions in the research process are "creating new roles and opportunities for libraries to help in preserving, managing, publishing, and accessing data."

Digital Preservation Matters.

Digital Stewardship in a Radio Archive: An NDSR Project Update

Digital Stewardship in a Radio Archive: An NDSR Project Update. Mary Kidd, Erin Engle. The Signal. January 5, 2016. 
     This is an update on the Radio Archive project and includes:

• a Digital Preservation Roadmap, 
• an overview of the current digital production throughout the various stations
• a network-wide analysis of NYPR’s digital holdings

The definition of what makes radio includes both traditional broadcasting as well as the digital formats, particularly the WAV and MP3 files, live and on-demand streaming audio and video, on-the-go podcasts, and social media posts. The archive will have to "develop new ways to address the fact that digital assets are often interconnected with other assets, rather than standalone audio objects".  Archives are not just safekeepers of the past, but informants of the present. The evolving identity of radio is influencing the development of a new and emerging field referred to as “radio preservation studies”.

An important part of the studies for the archive is to distinguish radio archives from traditional sound archives.They appear to be similar, but radio has its own distinct sound and modes of production. And radio archives require their own set of archival best practices that contextualize broadcast recordings. "Striking a balance between the archival priorities and the expectations of producers is quite possibly one of the greatest challenges for an archive embedded in a media station."

Digital Preservation Matters.

Research Software Sustainability: Report on a Knowledge Exchange Workshop

Research Software Sustainability: Report on a Knowledge Exchange Workshop. Simon Hettrick. The Software Sustainability Institute. February 2016; 3 March 2016.   [PDF]
     "Without software, modern research would not be possible since it is connected to the software that is used to generate results." Overlooking software will put at risk the reliability and the ability to reproduce the research itself. Like the research, and any other tool, software must stand up to the same scrutiny. It is not easy to define software sustainability, but it is the practices that allow software to continue to function as expected in the future.  This is neither easy nor straightforward. Software has a lifecycle: it is conceived, matures and decays. Not all software should be sustained, we should  concentrate on sustaining software that is most useful. Software is always reliant on other software in order to work., including operating system, system libraries, and other necessary packages. Any change or decay at any level can affect the operation of the software higher up the stack.  "If we attempt to preserve software, it quickly becomes out of step with its dependent software."

Definitions:

• Research software: software developed within academia for the purposes of research, particularly to generate, process and analyze results.
• Software sustainability: the technical and non-technical practices that allow software to continue to operate as expected in the future. A constant level of effort is required to maintain the software’s operation.
• Software preservation: an approach to extend the lifetime of software that is no longer actively maintained.
• Software archiving: one important aspect of software preservation. It is the process of storing a copy of software so that it may be referred to in the future.

Approaches to software sustainability and preservation

1. Encapsulation. Preserve the original hardware and software to ensure that the software continues to operate (an example is recomputation.org)
2. Emulation. Emulate the original hardware and operating environment so that the software continues to operate
3. Migration. Update the software to maintain the original functionality and transfer it to new platforms as necessary to prevent obsolescence
4. Cultivation. Keep the software up to date by adopting an open development model that allows new contributors to be brought on board
5. Hibernation. Preserve knowledge of how to resuscitate the software’s exact functionality at a later date
6. Deprecation. Formally retire the software. Unlike hibernation, no time is invested into preparations to make it easier to resuscitate the software
7. Procrastination. Do nothing

Key recommendations

1. Raise awareness of the fundamental role of software in research
2. Recognize research software as a valuable research object
3. Promote software sustainability
4. Embed software sustainability skills in the research community
5. Create organizations as focal points for software sustainability expertise

Benefits include:

1. Trusted research
2. Increased rate of discovery
3. Increased return on investment
4. Research data remains readable and usable

Digital Preservation Matters.

Leveraging Heritrix and the Wayback Machine on a Corporate Intranet: A Case Study on Improving Corporate Archives

Leveraging Heritrix and the Wayback Machine on a Corporate Intranet: A Case Study on Improving Corporate Archives. Justin F. Brunelle, et al. D-Lib Magazine. January/February 2016.
     This is a case study about using open-source, web-scale web archiving tools, Heritrix and the Wayback Machine. Internet archiving does not have the opportunity to archive Intranet-based resources, such as corporate content. Past research has shown that "web pages' reliance on JavaScript to construct representations leads to a reduction in archivability".  The Internet Archive uses Heritrix and the Wayback Machine to archive web resources and replay mementos on the public web.
The article recommends content authors use robots.txt and noarchive HTTP response headers to avoid sensitive information. Accidentally archiving sensitive information can result in loss of mementos within a WARC. Recommendations include:

• Use smaller storage devices to limit the problems if sensitive information is crawled;
• Develop a way to remove a sensitive memento from a WARC file 
• Identify high-risk vs. low-risk archival targets within the Intranet.

Archiving intranet content needs to fit within a larger documentation plan and knowing what the key resources and elements are that need to be preserved in order to preserve corporate memory. There is value for a corporation to have a web crawling archiving strategy. It "may make more sense for a corporate archives to preserve information about its corporation's projects that is tracked in a database and served to an Intranet through an export directly from the database rather than crawling the Intranet for the project data".

The case study and the next steps proposed will help archive corporate memory, improve information longevity, and can help corporate archivists implement web archiving strategies.

Digital Preservation Matters.

ODF: The Open Document Format

ODF: The Open Document Format. Carl Fleischhauer, Erin Engle. The Signal. January 19, 2016.
     In December 2015, descriptions of eleven new parts of the Open Document Format group were added to the Format Sustainability website. The Format Sustainability website provides technical descriptions about formats of all types to help staff in assessing new content. ODF is part of a complex format family, which exists in several versions and “parts.” The list of ODF-related formats added to the sustainability website last month:

•     ODF_Family, ODF (OpenDocument Format) Family, OASIS and ISO/IEC 26300
•     ODF_package_1_1, OpenDocument Package Format, ODF 1.1, ISO/IEC 26300:2006
•     ODF_package_1_2, OpenDocument Package Format, ODF 1.2 part 3; ISO/IEC 26300-3:2015
•     ODF_text_1_1, OpenDocument Text Format (ODT), ODF 1.1, ISO/IEC 26300:2006
•     ODF_text_1_2, OpenDocument Text Format (ODT), ODF 1.2, ISO/IEC 26300-1:2015
•     ODF_chart_1_2, OpenDocument Chart Format (ODC), ODF 1.2, ISO/IEC 26300-1:2015
•     ODF_draw_1_2, OpenDocument Drawing Format (ODG), ODF 1.2, ISO/IEC 26300-1:2015
•     ODF_spreadsheet_1_1, OpenDocument Spreadsheet Format (ODS), Version 1.1, ISO 26300:2006
•     ODF_spreadsheet_1_2, OpenDocument Spreadsheet Format (ODS), Version 1.2, ISO 26300:2015
•     ODF_dbfront_1_2, OpenDocument Database Front End Document Format (ODB), Version 1.2, ISO 26300-1:2015
•     ODF_presentation_1_2, OpenDocument Presentation Document Format (ODP), Version 1.2, ISO 26300-1:2015

ODF was designed so no one using public documents would be "forced to buy software from one particular vendor or for one particular operating system platform.” It has been supported by institutions trying to preserve documents for the long term. There are also a number of organizations, including government, that have "adopted the ODF family of formats as mandatory or recommended for documents that must be editable in order to support collaboration within the government or between the government and the public."

Digital Preservation Matters.

Back in a Flash

Back in a Flash. Edith Halvarsson. Open Preservation Foundation, ehalvarsson's Blog. 27 Jan 2016.
     Flashback is a proof of concept project run by the British Library’s Digital Preservation Team that examines emulation and migration solutions as methods for preserving the content on CD, DVD , 3.5” and 5.25” disks.  The team acquired original hardware for their legacy lab to analyze and deal with content from those formats. They have found that the old hardware can have problems. The first step is a capture process which extracts data from the storage media and characterizes its physical components and lists the files on the media. The content can be placed in a controlled environment that ensures that the bits are retained regardless of deteriorating storage media. The technical information about the content is important for preservation planning.

For less complex content such as text the solution is to migrate the file from for old or obsolete formats to  more contemporary and reliable formats. The large majority of the content though is so "tightly bound up with its original environment that it cannot be migrated", which is the case for software. For these, the option is to emulate the item’s original hardware and software environment which were supplied by the University of Freiburg via BwFLA – Emulation As A Service. Flashback is gathering data about the performance and viability of emulating groups and comparing characteristics of the software on original hardware and emulators. 

Digital Preservation Matters.

Having FITS Over Digital Preservation?

Having FITS Over Digital Preservation? Jeffrey Erickson. NDSR Boston. February 11, 2016.
     FITS (File Information Tool Set) is an open source digital preservation tool designed to identify and validate a wide assortment of file formats, determine technical characteristics, and extract embedded metadata. The technical metadata from FITS can be exported to XML schemas. Digital preservation repositories contain a growing number of file formats. "Proper identification of a file’s format and the extraction of embedded technical metadata are key aspects of preserving digital objects. Proper identification helps determine how digital objects will be managed and extracting embedded technical metadata provides information that future repository staff or users need to render, transform, access and use the digital objects." The current version of FITS bundles many other tools together, and makes them all easier to use; some are: Droid; ExifTool; ffident; Jhove; MediaInfo (video files); New Zealand Metadata Extractor Tool. Using multiple tools can help verify the file information.

FITS consolidates and normalizes the output, providing a homogenized data set that is easier to interpret. The output can be inserted into other files, such as METS files, that can provide digital preservation documentation about the file.  FITS can assist with quality control, improving metadata, format migration metadata. FITS Sites: fitstool.org  GitHub

Digital Preservation Matters.

Where in the Org Chart is Digital Preservation?

Where in the Org Chart is Digital Preservation? Shannon Virginia Zachary. Bits and Pieces blog. January 21, 2016.
     Where on the organizational hierarchy does a digital preservation fit in a research library:IT, collections management, or preservation? Often it is bundled with other preservation responsibilities,  digital creation, curation, and delivery tasks. Early on, "digital activities were often assigned to a specific person, probably in a pilot project", then collected into a single department.  As the programs grew, "policies and practices for preservation were developed in silos—or not at all in the intense focus on creation and delivery." For the library, there were questions about creating a Digital Preservation Position; they realized the need for a position with responsibility for the development and management of preservation policies. Questions that need to be answered are, where the position should reside; in the job a management or a technical job; and is it a digital library responsibility or a library responsibility?  Discussions with stakeholders made it sound like it belongs in preservation; it needs to work with others throughout the library, and policies and technical solutions need to be developed to make preservation happen.

• Above all, preservation requires communication and partnership between those with specialized technical knowledge and those with collection knowledge and responsibility: how the collection grows, how it is used, what materials are likely to be wanted ten years from now or fifty or a hundred. 
• The preservation specialist needs to be able to talk management with managers and technicalities with technical experts.

Where digital preservation work is located on the org chart doesn’t matter as long as the "library recognizes that preservation needs to happen and that responsibility for preservation must be assigned somewhere". Recognizing the similarities in digital collections with what libraries have always done, however, will provide a strong foundation for sustainability.

Digital Preservation Matters.

Preserving Social Media

New Technology Watch report: Preserving Social Media. Sara Day Thomson. Digital Preservation Coalition and Charles Beagrie Ltd. 16 Feb 2016. [PDF]
     This report looks at the related issues of preserving social media. Institutions collecting this type of media need new approaches and methods.  The report looks at "preserving social media for long-term access by presenting practical solutions for harvesting and managing the data generated by the interactions of users on web-based networking platforms such as Facebook or Twitter." It does not consider blogs. "Helen Hockx-Yu defines social media as: ‘the collective name given to Internet-based or mobile applications which allow users to form online networks or communities’.

Web 1.0 media can be harvested by web crawlers such as Heritrix; Web 2.0 content, like social media platforms, is more effectively archived through APIs. This is often an extension of an institution's web archiving. Transparency and openness will be important when archiving content. APIs allow developers to call raw data, content and metadata directly from the platform, all transferred together in formats like JSON or XML.

Maintaining long-term access to social media data faces a number of challenges, such as working with user-generated content, continued access to social media data, privacy issues, copyright infringement issues, and having a way to maintain the linked, interactive nature of most social media platforms. There is also "the challenge of maintaining the meaning of the social media over time, which means ensuring that an archive contains enough metadata to provide meaningful context."  There are also third-party services and self-archiving services available.

Social media is vulnerable to potential loss. The report quotes one study which looked at "the lifespan of resources shared on social media and found that ‘after the first year of publishing, nearly 11% of shared resources will be lost and after that we will continue to lose 0.02% per day’."

Some other quotes:

• Overall, the capture and preservation of social media data requires adequate context.
• Capturing data, metadata, and documentation may not provide enough context to convey user experiences with these platforms and technologies.
• When considering the big picture, however, the preservation of social media may best be undertaken by a large, centralized provider, or a few large centralized providers, rather than linking smaller datasets or collections from many different institutions.

Digital Preservation Matters.

Filling the Digital Preservation Gap

Filling the Digital Preservation Gap. A Jisc Research Data Spring project; Phase Two report. Jenny Mitcham, et al. February 2016. [PDF]
     The report is a collaboration between the universities of Hull and York and during phase 2 of the project they worked closely with Artefactual Systems to enhance Archivematica to enable it to work better as part of a wider system for managing and preserving research data. The report also includes implementation plans for establishing Archivematica at each institution.  (Phase 1 of the project (Filling the Digital Preservation Gap. A Jisc Research Data Spring project. Phase One report July 2015. Jenny Mitcham, et al.) looked at digital preservation as part of a wider infrastructure for research data management.)

Some items of interest from the report:

• In order to manage research data effectively for the long term we need to consider how we incorporate digital preservation functionality into our Research Data Management workflows.
• research datasets can be large, of mixed formats and their value may not be fully understood. 
• Creating access copies may be unnecessary as some datasets will never be requested for reuse.
• One of the potential bottlenecks in the current Archivematica pipeline is checksum generation
• populating PRONOM is not a oneoff exercise. We need to find ways to continue to engage and submit samples in order that new file signatures can be created as the need arises.

The report also included a glossary and an appendix discussing "Hydra in Hull Preservation workflows"

Digital Preservation Matters.

Avoid Jitter! Measuring the Performance of Audio Analog-to-Digital Converters

Avoid Jitter! Measuring the Performance of Audio Analog-to-Digital Converters. Carl Fleischhauer; Erin Engle. The Signal. February 19, 2016.    
     An article on audio recordings and conversion to digital files.  There are references to  a revision of a FADGI guideline (Audio Analog-to-Digital Converter Performance Specification and Test Method, Feb 2016) and an accompanying explanatory report (ADC Performance Testing Report on Project Development During 2015. Feb 2016.) Audio tapes deteriorate over time and some institutions are digitizing the files in order to save the recorded content.

Most preservation specialists recommend that the waveform be sampled 96,000 times per second (96 kHz) which will capture the “horizontal” frequency of the wave movement. To capture the amplitude, the “vertical” movement, each sample should be 24 bits long. This can be a challenge due to the many factors involved.

Digital Preservation Matters.

When Your Floppies Flop, Make Them Digital At DCPL's Memory Lab

When Your Floppies Flop, Make Them Digital At DCPL's Memory Lab. Rachel Kurzius. DCist. Feb 18, 2016.
     A new resource in the Digital Commons at the Martin Luther King Jr. Memorial Library gives people the technology and know-how to digitize their personal collections. Changing hardware and software make formats difficult to use, such as floppy disks, audio cassettes, and VHS tapes. The Memory Lab will be developing a curriculum that teaches people how to preserve their digital history. "We want to have some slice of the culture that's happening now for the future. As an archivist, we've been really concerned that people's personal collections are changing and are no longer just physical." They believe it is "the only project of its kind in the country with a specific eye on stewardship and preservation." The preservation actions are "mainly about how to name files, add descriptions, store them in the best possible format and location, and how to be an informed consumer."

Digital Preservation Matters.

Eternal 5D data storage could record the history of humankind

Eternal 5D data storage could record the history of humankind. Press Release. Optoelectronics Research Centre, University of Southampton. February 16, 2016.
     Scientists have developed the recording and retrieval processes of five dimensional (5D) digital data processing with femtosecond lasers that may be capable of  storing digital data for billions of years on nanostructured glass.  "The storage allows unprecedented properties including 360 TB/disc data capacity, thermal stability up to 1,000°C and virtually unlimited lifetime at room temperature (13.8 billion years at 190°C )".  This encoding on ‘Superman memory crystal’ is in five dimensions: the size and orientation in addition to the three dimensional position of these nanostructures.

Related:

• Researchers Raid the Fifth Dimension for ‘Eternal Data Storage’

Digital Preservation Matters.

A Digital ‘Library of Alexandria’

A Digital ‘Library of Alexandria’. Katie McNally. UVAToday. February 10, 2016.
     "Scholars often lament the knowledge that might have been preserved if the great Library of Alexandria had been better protected."  Digital collections face a similar threat of "steady extinction" because of technological obsolescence. One safeguard is the Academic Preservation Trust (APTrust) which is "a large-scale solution that preserves digital scholarship by storing it across multiple technologies and physical locations." The primary goal is to package and preserve information in a way so it will be accessible to future generations.  Besides proper description, “Deep dark preservation” refers to all the pieces needed to "effectively archive a digital file and the technology it runs on for future use."  So the digital preservation is really a phased thing: address those items which can be done quickly, then work on the more difficult problems, such as finding ways to preserve the software that makes digital objects accessible. Emulation environments are being worked on to keep old software running. 

APTrust stores files in two separate Amazon data centers, one in Virginia and one in Oregon, and each of these use different technologies to store the data, to help protect against the "failure of future and modern technologies.” [APTrust is also one of the DPN nodes.]

Digital Preservation Matters.

Presentations From LC/NARA Symposium on Archiving Email Made Available Online

Presentations From LC/NARA Symposium on Archiving Email Made Available Online. Gary Price. Library Journal. February 2, 2016. [Video and text].
      Videos of six presentations from the “Archiving Email Symposium” held at the Library of Congress June 2015 are now available online. This was a symposium of federal agencies, academic and research libraries, technologists, curators, archivists and records managers who are directly working on collecting and preserving email archives in order to discuss challenges and solutions. The videos are embedded in the article and links point to transcripts.

Archiving Email: Welcome & Introduction  [text]

• Mark Sweeney (Library of Congress)
• Paul Wester (National Archives and Records Administration)

Institutional Approaches to Archiving Email  [text] 

• Ricc Ferrante (Smithsonian Institution Archives)
• Lynda Schmitz Fuhrig (Smithsonian Institution Archives)
• Jaime Schumacher (Northern Illinois University and Digital POWRR Project)

Challenges of Email as a Record  [text]  

• Lisa Haralampus (U.S. Office of the Chief Records Officer)
• Deborah Armentrout (Nuclear Regulatory Commission)
• Jeanette Plante (U.S. Department of Justice)
• Edwin McCeney (U.S. Department of the Interior)

Policy Development for Archiving Email  [text] 

• David Kirsch (University of Maryland)
• Anthony Cocciolo (Pratt Institute School of Information and Library Science)
• Kenneth Hawkins (National Archives and Records Administration)
• Kathleen O’Neill (Library of Congress)
• Margaret McAleer (Library of Congress)
• Christopher Hartten (Library of Congress)

Practical Approaches to Processing Email  [text]  

• Roger Christman (Library of Virginia)
• Aprille Cooke McKay (University of Michigan)
• Dorothy Waugh (Emory University)

Archiving Email: Closing Summary   [text] 

• Chris Prom (University of Illinois Urbana-Champaign)

Digital Preservation Matters.

To ZIP or not to ZIP, that is the (web archiving) question

To ZIP or not to ZIP, that is the (web archiving) question. Kristinn Sigurðsson. Kris's blog. January 28, 2016.
     This post looks at the question: Do you use uncompressed (W)ARC files? Many files on the Internet are already compressed and there is "little additional benefit gained from compressing these files again (it may even increase the size very slightly)."  For other files, such as text, tremendous storage savings can be realized using compression, usually about 60% of the uncompressed size. Compression has an effect on
disk or network access and on memory. But "the additional overhead of compressing a file, as it is written to disk, is trivial."

On the access side, the bottleneck is disk access but "compression can actually help!" It can save time and money and performance is barely affected. One exception may be with HTTP Range Requests which when accessing a WARC record would have to decompress the entire payload until it finds the requested item. A hybrid solution may be the best solution: "compress everything except files whose content type indicates an already compressed format."  This would also avoid a lot of unneeded compression / uncompression.

Digital Preservation Matters.

“High-res audio”

“High-res audio”. Gary McGath. Mad File Format Science Blog. February 8, 2016.
    High-res audio, sound digitized at 192,000 samples per second is not necessarily better than the usual 44,000. We can only hear sounds only in a certain frequency range, generally 20 to 20,000 Hertz.

"The sampling rate of a digital recording determines the highest audio frequency it can capture. To be exact, it needs to be twice the highest audio frequency it records." Delivering playback audio at a higher rate offers no benefit and may introduce problems. Another important part of audio is the number of bits per sample, usually 16 bits, but higher-res audio often offers 24 bits. This "isn’t likely to cause any problems", but it doesn't necessarily provide a benefit.   A bigger problem is "over-compressed and otherwise badly processed files".  It is important to not skimp on quality.

Digital Preservation Matters.

Keep Your Data Safe

Love Your Data Week: Keep Your Data Safe. Bits and Pieces.  Scott A Martin. February 8, 2016.
     The post reflects on a 2013 survey of 360 respondents:

• 14.2% indicated that a data loss had forced them to re-collect data for a project.  
• 17.2% indicated that they had lost a file and could not re-collect the data.

If this is indicative of  the total population of academic researchers, then there is a lot of lost research time and money due to lost data. Some simple guidelines can greatly reduce the chances of catastrophic loss if steps are included in your own research workflow:

1. Follow the 3-2-1 rule for backing up your data: store at least 3 copies of each file (1 working copy and 2 backups), 2 different storage media and at least 1 offsite copy 
2. Perform regular backups
3. Test your backups periodically
4. Consider encrypting your backups.  Just make sure that you’ve got a spare copy of your encryption password stored in a secure location!  

Digital Preservation Matters.

New digital preservation solution from Arkivum

New digital preservation solution from Arkivum, shaped to grow with your data. Nik Stanbridge. Arkivum Press release. January 21, 2016.
     Arkivum is launching a new cloud-based digital preservation and archiving service with Artefactual Systems Inc. of Vancouver. "Arkivum/Perpetua is a cost-effective, comprehensive, fully hosted and managed digital preservation and public access solution that uses Archivematica and AtoM (Access to Memory) services in the cloud."

In a survey of archivists and data curators, 87% said "file format preservation and data integrity were important elements to their digital preservation workflow. And a third of respondents stated that they would be using a cloud-based solution for their digital preservation data."

Digital Preservation Matters.

MRF for large images

MRF for large images. Gary McGath. Mad File Format Science Blog. January 21, 2016.
NASA, Esri speed delivery of cloud-based imagery data. Patrick Marshall. GCN. Jan 20, 2016.
     NASA and Esri are releasing to the public a jointly developed raster file format and a compression algorithm designed to deliver large volumes of image data from cloud storage.  The format, called MRF (Meta Raster Format) together with a patented compression algorithm called LERC, can deliver online  images ten to fifteen times faster than JPEG2000. The MRF format breaks files into three parts which can be cached separately. The metadata files can be stored those locally so users can "examine data on file contents and download the data-heavy portions only when needed". This would help to minimize the number of files that are transferred. The compression allows users to get faster performance, lower storage requirements, and they estimate the cloud storage costs would be about one-third as much as traditional file-based enterprise storage. An implementation of MRF from NASA is available on GitHub and an implementation of LERC is on GitHub from Esri.

Digital Preservation Matters.

Developing a Born-Digital Preservation Workflow

Developing a Born-Digital Preservation Workflow. Jack Kearney, Bill Donovan. April 8, 2014.
     Presentation that looks at developing a systematic approach to preserving digitally born collections. The example from Boston College are the Mary O’Hara papers. This was an opportunity for a collaborative project involving the Digital Libraries, Archives, and the Irish Music Center.

• Important elements of the workflow:
• Chain of Custody, 
• Digital Forensics, 
• Computed initial checksums, 
• File/folder names, 
• Local Archival Copies, Distributed Digital Preservation

“Digital forensics focuses on the use of hardware and software tools to collect, analyze, interpret, and present information from digital sources, and ensuring that the collected information has not been altered in the process.” The presentation has some specific steps and procedures in ways to not alter the information, including multiple copies, write blockers, and such. In working with external drives, they would build and output an inventory taken with this Unix command:
     :  find directory-name  -type f -exec ls -l {} ; >c:\data\MOH\inventory.txt

Local conventions regarding naming files and folders:

• Use English alphabet and numbers 0 - 9
• Avoid punctuation marks other than underscores or hyphens.
• Do not use spaces.
• Limit file/folder names to 31 characters, including the 3 digit extension . Prefer shorter names.
• Decision: They may remediate folder and file names, but only for the working copies.

They also look for files that need actions taken:

• Any files off-limits or expendable? System files,
• Personally Identifiable Information (PII)
• Unsupported Formats (Can normalize using Xena)
• They also use a variety of tools, such as: FITS,  JHove 

Important to keep track of digital preservation actions:

• File migrations
• Obsolete file formats
• Proprietary file formats
• Metadata changes

Digital Preservation Matters.