Using Open-Source Tools to Fulfill Digital Preservation Requirements

OSS4EVA: Using Open-Source Tools to Fulfill Digital Preservation Requirements. Marty Gengenbach, et al. Code4Lib. 2016-10-25.
     Open-source software has played an increasingly prominent role in digital preservation, such as LOCKSS, DSpace, and DROID. The number and variety of such tools has increased, there was a growing need among preservationists to assess how and when to adopt particular tools so that they could better support their institutions’ specific requirements and workflows.  Open-source projects allows the user community to contribute by developing and documenting tools.

There are some challenges with open source programming.

• Perceptions of instability:  One challenge is the perception that these tools are "inherently unstable and therefore present a risk". 
• Resources and funding: Administrators often are reluctant to commit resources to an open source project. Funding problems can threaten the long-term sustainability of open source tools.
• System updates: Open source tools require regular patches, updates, and upkeep. Without this, the tool would be outdated, and open to security holes. "The choice to maintain an unsupported version of a particular open-source tool simply because it meets (or has been customized to meet) an organization’s needs is problematic. For what an institution may stand to gain from this tool in terms of functionality and local integration, it may stand to lose in terms of the stability of a mainstream code release, the risk to information security, and the likelihood that the tool in question will become increasingly less functional and reliable as it ages".
• Integration. Integrating open-source tools into institutional workflows can be a challenge, taking into account software dependencies, system requirements, and local configuration to put the tools into a production environment. This can require a considerable time and resources. 

One of the possible benefits is that institutions can customize open source tools for use within a specific context, but that comes with its own hurdles, such as reducing the ability to draw on the user community.  The digital preservation open source landscape has evolved from a scattered set of standalone tools designed to complex software environments. "Nevertheless, these tools still are not watertight." There are real concerns about open-source tools that can pose serious risks to collections.

Digital Preservation Matters.

Open Source Tools for Records Management

Open Source Tools for Records Management. National Archives and Records Administration. March 18, 2015. [PDF, 22pp.]
      NARA has identified open source tools that could be used for records management, but it does not include proprietary free software tools. Security is a concern with some implementations of open source tools.  These are  neither tested nor endorsed by NARA. The list of tools is approximately 18 pages; the tools address functionality, such as:
managing workflows; identifying duplicates; extracting and managing metadata; handling email archives; web publishing; data analyzing; working with PDF files; preservation planning; scanning files; identifying confidential data; file renaming; web archiving; comparing web pages; document managing; format identification; file integrity; image processing; and natural language processing.

The document also contains lists of other resources and tools.

Related posts:

• Digital Curation Tools & Techniques
• Lessons learned in developing digital preservation tools...
• Digital Tools and Apps 
• Digital Preservation Tools
• All posts on tools
  

Digital Preservation Matters.

Digital Tools and Apps

Digital Tools and Apps. Chris Erickson. Presentation for ULA. 2014. [PDF]
This is a presentation I created for ULA to briefly outline a few tools that I find helpful. There are many useful tools, and more are being created all the time. Here are a few that I use.

• Copy & Transfer Tools: WinSCP; Teracopy;
• Rename Tools: Bulk Rename Utility
• Integrity & Fixity Tools: MD5Summer; MD5sums 1.2; Quick Hash; Hash Tool
• File Editing Tools: Babelpad; Notepad++; XML Notepad; 
• ExifTool; BWF MetaEdit; BWAV Reader;
• File Format Tools: DROID; 
• File Conversion:  Calibre; Adobe Portfolio;
• Others: A whole list of other tools that I use or suggest you look at.
•  PDF/A tools
• Email tools

 Please let me know what tools you find helpful.

Digital Preservation Matters.

Digital Preservation Tools.

Digital Preservation Tools.  Paul Wheatley. SPRUCE Project Wiki. May 10, 2012.
Open Planets Foundation Digital Preservation Tool Registry. This lists many tools of all types that may be just what you are looking for, and also references to additional tool lists, such as http://agogified.com/tools-and-services.  A few of the categories include:

• Transcription Tools
• DataOne
• Forensic tools
• PDF Creation
• File Conversion tools
• Audio file conversion tools
• Web capture
• Email mining tools
• Record analysis tools
• Curation and data management tools
• Bulk extractor tools
• etc.

Digital Preservation Matters. 

Introduction to Metadata Power Tools for the Curious Beginner

Introduction to Metadata Power Tools for the Curious Beginner. Maureen Callahan, Regine Heberlein, Dallas Pillen. SAA Archives 2015. August 20, 2015.   PowerPoint  Google Doc 
      "At some point in his or her career, EVERY archivist will have to clean up messy data, a task which can be difficult and tedious without the right set of tools." A few notes from the excellent slides and document:

Basic Principles of Working with Power Tools

• Create a Sandbox Environment: have backups. It is ok to break things
• Think Algorithmically: Break a big problem down into smaller steps
• Choosing a Tool: The best tools, works for your problem and skill set
• Document: Successes, failures, procedures

Dare to Make Mistakes

• as long as you know how to recognize and undo them!
• view mistakes as an opportunity
• mistakes can teach you as much about your data as about your tool
• share your mistakes so others may benefit
• realize that everybody makes them

General Principles

• Know the applicable standards
• Know your data
• Know what you want
• Normalize your data before you start a big project
• The problem is intellectual, not technical
• Use the tools available to you
• Don’t do what a machine can do for you
• Think about one-off operations vs. tools you might re-use or re-purpose
• Think about learning tools in terms of raising the level of staff skill

Tools

• XPath
• Regex
• XQuery
• XQuery Update
• XSLT
• batch
• Linux command line
• Python
• AutoIt

Digital Preservation Matters.

Some Assembly Required – Micro-services and Digital Preservation

Some Assembly Required – Micro-services and Digital Preservation. Danielle Spalenka. POWRR Blog. March 22, 2016.   
     Very informative article about how micro services and tools can benefit libraries of all sizes and financial abilities. Many struggle with implementing a digital preservation infrastructure.  University of California created a set of free-standing but inter-operable applications that performed a single or limited number of tasks in the larger curation and preservation process, which they described as a micro-services approach.

This approach, which did not require the installation of a single, long-lived application, can help medium-sized and smaller institutions to identify and achieve digital preservation goals. The set of twelve independent but compatible micro-services performed preservation functions such as identity, storage, fixity, replication, inventory, ingest, index, search, transformation, notification and annotation. These simple utilities would pose fewer challenges in their development, deployment, maintenance and enhancement than a large, integrated system. The strategic combination of individual services could produce “the complex global function needed for effective curation” at large institutions. The Digital POWRR (Preserving Digital Resources with Restricted Resources) Project team begin a study of the problems and possible solutions for preserving digital objects.

Some understood that digital curation and preservation was an either/or issue: either an institution had implemented a digital preservation system or it had not. In reality, the preservation activities are "an ongoing, iterative set of actions, reactions, workflows, and policies." This means institutions can begin taking small steps rather than waiting to devise an ideal solution. The NDSA Levels of Digital Preservation provides a yardstick to measure progress toward a digital curation and preservation capacity. Two fundamental understandings at the heart of a micro-services approach are:

1. digital curation and preservation is an uncertain process in which continuous, rapid technological change often renders monolithic, integrated applications cumbersome and outdated;
2. simple tools focused on a specific aspect or aspects of the process can prove more helpful.

Micro-services tools can only be effective if users understand what roles they play in the larger digital preservation process and the path they take through the NDSA Levels. The COPTR (Community Owned Digital Preservation Tool Registry) web site provides information about many helpful tools and services that can provide incremental value.  Micro-services tools can also help those adopting more robust tools.

The use of individual tools performing discrete functions can help those starting preservation activities. The Digital POWRR Project has described the stages of a digital curation and preservation workflow and associated activities. Tools performing these functions are available at the COPTR web site.

Digital Preservation Matters.

Digital Preservation Matters - 09 September 2009

Harvard's Web Archive Collection Service (WAX). Website. September 2009.

This site began as a pilot project to address the management of web sites by collection managers for long-term archiving. It is designed to capture, manage, store and display web sites in an archive. “With blogs supplanting diaries, e-mail supplanting traditional correspondence, and HTML materials supplanting many forms of print collateral, collection managers have grown increasingly concerned about potential gaps in the documentation of our cultural heritage.” It is built using open source tools such as the Heritrix web crawler; the Wayback index and rendering tool; and the NutchWAX index and search tool. Documents concerning WAX are available at: http://hul.harvard.edu/ois/support/docs-wax.html

Glossary of Web Archiving Terms. Molly Bragg. Internet Archive website. August 06, 2009.

Part of the Archive-It help section. This page has a glossary of web archiving terms. The rest of the wiki has some good information about archiving web sites.

Missing links: the enduring web. Web Archiving Consortium Workshop. 21 July 2009.

Web pages are at risk. The early web pages are of similar historical importance with prehistoric writings, and both are at risk. “Key issues for long-term access and preservation remain unresolved.” This site includes the presentations from the workshop. Some of these include:

• Web Archive and Citation Repository in One: DACHS
• The future of researching the past of the Internet
• Web Archiving Tools: An Overview
• Context and content: Delivering Coordinated UK Web Archive to User Communities
• Capture and Continuity: Broken links and the UK Central Government Web Presence
• Diamonds in the Rough: Capturing and Preserving Online Content from Blogs
• Beyond Harvest: Long Term Preservation of the UK Web Archive
• From Web Page to Living Web Archive
• Emulating access to the web
• What we want with web-archives; will we win?

The following items are a few of the presentations at the Web Archiving Consortium Workshop:

Web Archive and Citation Repository in One: DACHS. Hanno Lecher.

Goals:

1. Capturing and archiving relevant resources as primary source for later research
2. Providing citation repository for authors and publishers

When citing online resources:

• Verify URL references
• Evaluate reliability of online resources
• Use PURLs
• Tools include: Snagit, Zotero, WebCite, DACHS Citation Repository

“the best current solution to improve access to Internet references is for publishers to require capture and submission of all Internet information at the time of manuscript consideration“

The future of researching the past of the Internet. Eric T. Meyer.

May not want to capture an entire web site, so you may consider the sub links. ‘Seed’ is a site from which other sites can be discovered through the links. Look at annotating the web sites; moving from snapshots of a site to more continuous data capture; how to share the results in a meaningful way.

Web Archiving Tools: An Overview. Helen Hockx-Yu.

1. Selection: have a policy, decide what to capture.
2. Collect data files (snapshot), examine for other sites to be collected, add to collection list
3. Store the archived files on disk, virus check, integrity check
4. Make accessible, index, add metadata, render the files, ensure long term access

Heretrix is the most commonly used tool, also Web Curator Tool.

WebARCive (WARC) format is coming into use. Other tools needed:

• Rendering, such as Open Source WaybackMachine;
• Full-text search, such as Nutch/Nutchwaxby or Hanzo tools
• Provide other search/retrieval options (subject, collection, site name, change over time)

No consensus on strategy, practices and specific tools. Crawlers work with HTML, but not advanced designs or tools. Need to handle problem sites. Decide what to duplicate.

Context and content: Delivering Coordinated UK Web Archive to User Communities. Cathy Smith.

The presentation starts with two questions:

1. What audiences should web archives anticipate and what does this mean for selection, ingest and preservation?
2. What will the web be like as an historical source, and what use will be made of archived web sites by future generations?

Recommendations:

1. Institutions continue to provide access to their individual collections, where appropriate, to support researchers; assure integrity of collections; allow integration with the institution’s other, non-web holdings;
2. Coordinate with other institutions by sharing collection development policies; defining the metadata standard, and developing technical interfaces.

Diamonds in the Rough: Capturing and Preserving Online Content from Blogs. Richard Davis.

“New genres of publications are becoming increasingly important to participants. For example, blogs are cited as a good window into what expert practitioners are doing. This material is not duplicated in traditional sources, yet it is important to consult”. Perceived barriers of web archiving are the cost of implementation and the complexity of available tools. Institutional blog archives are part of the institutional record. They should go through a selection process; support authenticity and fixity; be persistent and citable. Blogs seem to be an area where the content is of primary importance and design is secondary. Create an institutional or thematic archive by using a WordPress database to gather and store the posts and comments and provide access.

Lessons learned in developing digital preservation tools the right way (and the wrong way).

Lessons learned in developing digital preservation tools the right way (and the wrong way). Paul Wheatley.  iPRES 2014 - Proceedings. October 2014.
There have been difficulties with creating digital preservation tools in the past, including poor technology choices, half measure in adopting open source applications, limited project funding, gaps in capabilities, poor support, and others. The wish list of tools from several years ago hasn't changed much. "Why can't we have digital preservation tools that just work?" But there have been new approaches which focus more on practical applications, sharing ideas with others and engaging with existing projects.

Lessons learned  in developing digital preservation tools:

• Be agile in the development. Develop, demo, and get feedback from users. Get crude results first, then perfect and polish later.
• Re-use, don't re-invent. There are often tools outside of the preservation community.Try existing solutions first.
• Keep it small and simple. Modularize in the face of growing requirements. Make it so the tool can be integrated into others' workflows.
• Make it easy to use, build on, re-purpose, and maintain. Share the source and package for easy install.
• Share results, knowledge; learn from others.Let others know what you are doing.

The poster includes:

• Engage with the community
• Build on existing work
• Design for longevity
• Ally with a more long lived organization as a custodian

Digital Preservation Matters.

Digital Curation Tools & Techniques

Digital Curation Tools & Techniques. Nancy McGovern.  Webinar: Metropolitan New York Library Council. April 2015.
This is a recording of Nancy's excellent webinar and it goes along with the management tools website.  As organizations build sustainable digital preservation programs, they work toward being a trusted digital repository by following emerging digital preservation standards. There are now a number of tools to help with technical issues and now some management tools to help with organizational issues. The Digital Preservation Management workshop launched a set of management tools and techniques.

This webinar looks at using a set of digital curation and preservation principles to describe a  framework for developing a digital preservation program. The organizational aspects of digital preservation is more challenging than the technological issues. This helps people to work with tools and techniques, develop good practices, emphasize the organizational aspects and apply the 5 stages:

1. Acknowledge: understanding that digital preservation is a local concern
2. Act: initiating digital preservation projects
3. Consolidate: segueing from project to basic program
4. Institutionalize: rationalizing local efforts to establish a comprehensive program
5. Externalize: embracing inter-institutional collaboration and dependency

Each tool and technique is built on the DPM model, the three-legged stool (Organizational Leg, Technological Leg, and Resources Framework) with the five stages of development for a sustainable digital preservation program. There is a tool or technique to help organizations in each of these areas:

1. Principles: Adopt standards-based principles 
2. Policy: Develop a high-level policy framework 
3. Scope: Complete a digital content review to define program scope
4. Workflow: Document workflows to improve and automate 
5. Preparedness: Extend disaster preparedness to include digital
6. Self-assessment: Engage in self-assessment to gauge progress

Digital Preservation Matters.

TIMBUS Project Web Portal: a Gateway to the TIMBUS Tools

TIMBUS Project Web Portal: a Gateway to the TIMBUS Tools. Timbus Project website. December 19, 2014.

The EU-cofunded TIMBUS project focuses on resilient business processes making the data accessible over long periods. Continued accessibility is often considered as a set of activities carried out in the isolation of a single domain. TIMBUS, however, considers the dependencies on third-party services, information and capabilities that will be necessary to validate digital information in a future usage context. TIMBUS will deliver activities, processes and tools that ensure

• continued access to services and software
• to produce the context within which information can be accessed, properly rendered, validated and transformed into knowledge.

This approach extends traditional digital preservation approaches by introducing the need to analyse and sustain accessibility to business processes and the supporting services, and it aligns preservation actions more fully with enterprise risk management (ERM) and business continuity management (BCM).  The complexity and scale of enterprise business processes makes TIMBUS exceptionally relevant.

This website is a gateway to the outputs of the 4-year TIMBUS Project. It focuses on materials that bridge the gap between the complex research carried out by organizations and industries looking to implement direct, usable approaches to the digital preservation of their business processes. The site contains:

• Tools to collect relevant information from software and systems to generate a picture of a whole network of processes.
• Legalities Lifecycle Management tools and training
•  Risk assessment tools and recommendations about collected process data 
• Digital Preservation Expert Suite which includes tools to gather risk assessment  to provide Digital Preservation as a possible solution.
  
  
  Digital Preservation Matters.
  
   

FW: Digital Preservation Matters - 13 August 2009

File Information Tool Set (FITS). August 6, 2009.

With the increase of digital projects that introduce new formats, it is increasingly important to have tools that deal with issues such as file format identification, validation and metadata extraction tools. FITS, developed by Harvard, acts as a wrapper for some existing tools, including JHOVE, Exiftool, the National Library of New Zealand Metadata Extractor, DROID, Ffident, and two original tools: FileInfo and XmlMetadata. The files can identify a file with a single result, or in the case of a conflict, can handle it in several ways. It is written in java and can be run from a command line or an interface. It is available for download and has a user guide.

Research Data Preservation and Access: The Views of Researchers. Neil Beagrie, et al. Ariadne. July 2009.

Data is becoming more central to interdisciplinary projects and has grown in size and complexity. This study tries to assess the feasibility and costs of developing and maintaining a shared digital research data service. It shows, with text and graphs, the disciplines where research data issues were of greatest concern, the storage features that are needed most, the retention period for data once the projects have ended, and how the data is shared. University managers have serious concerns about the cost, scalability and sustainability of purely local solutions.

Library of Congress Digital Preservation Newsletter. August 2009.

LC has developed new tools (including bagit) to transfer large quantities of digital content. BagIt, and related transfer tools, prepare to transfer data by packaging the collection in a directory with a manifest file that lists the contents. Specifications and other tools are on the tool and services page. More on this: 21st Century Shipping. D-Lib Magazine. Michael Ashenfelder. July/August 2009

The California Digital Library has opened its Web Archiving Service collections. The service was created to support the Web-at-Risk project, and is funded by the NDIIPP and the University of California.

A workshop on photometadata aimed at helping digital photographers use metadata when creating and distributing their work. The program demonstrated applications to embed metadata in photographs; it was stated that each digital photo can and should contain information about itself, its creator and its licensing conditions. Industry professionals told how metadata increased their business.

Online textbooks are gaining popularity, changing how students study. Dani Martinson. Missourian. August 6, 2009.

Online textbooks can provide additional information and resources for students, including direct links to audio and video. Digital textbooks are usually 50%cheaper than regular textbooks, though there is no buyback, and the books are often available only for a semester. Information can be updated easier and more frequently. A study found that the professors were more accepting of digital textbooks than students. They expect the demand will increase when the digital content is specifically designed for digital, rather than just a PDF version of the printed textbooks.

Autonomous Preservation Tools in Minimal Effort Ingest

Autonomous Preservation Tools in Minimal Effort Ingest. Asger Askov Blekinge, Bolette Ammitzbøll Jurik, Thorbjørn Ravn. Andersen.  Poster, iPres 2016.  (Proceedings p. 259-60 / PDF p. 131).
     This poster presents the concept of Autonomous Preservation Tools developed by the State and University Library, Denmark. It is an expansion of their idea of Minimal Effort Ingest. In Minimal Effort Ingest most of the preservation actions are handled within the repository when resources are available. The incoming data is to be secured quickly, even when resources are sparse. Preservation actions should happen when resources are available, rather than by a static ingest workflow.

From these concepts they created the idea of Autonomous Preservation Tools which are more like software agents rather than a static workflow system. The process is more flexible and allows for easy updates or changes to the workflow steps. A fixed workflow is replaced with a decentralised implicit workflow which defines the set of events that an AIP must go through.  Rather than a static workflow that must process AIPs in a fixed way, the Autonomous Preservation Tools "can discover AIPs to process on their own". Because AIPs maintain an record of past events tools can determine whether or not an AIP has been processed or if other Tool actions must be performed first. So the workflow is the tools finding and processing items correctly until every item has been processed.  This becomes an alternative method of processing.

Digital Preservation Matters.

Digital Archives & Preservation Systems

Digital Archives & Preservation Systems. Kari R. Smith.  MIT Institute Archives. Engineering the Future of the Past Blog. July 1, 2015. [PDF]
     Very good presentation on fitting tools, processes and systems together. These open source tools are being developed and used in the digital archives and digital preservation communities to address the needs and gaps in systems for meeting standards, such as OAIS, PAIMAS, and Trustworthy Digital Repositories. They enable description, discovery, delivery, and preservation of digital collections. Some of the tools mentioned in the presentation:

• ArchivesSpace: archives collection management and discovery
• Archivematica: digital preservation system (processes files chosen for preservation and creates Archival Information Packages and Dissemination Information Packages)
• Access to Memory (atom): archives and special collections discovery and delivery system and collections management system
• BitCurator and BitCurator Access: digital forensics tools and processes for archival and special collections material
• ePADD: electronic mail processing, access, discovery system
• Fixity: enables the preservation actions of fixity checking of files in archival (or other) storage. Includes reporting on file attendance and file integrity.
• QuickView Plus: render the files so you can appraise them and make decisions
• Binder: integrates with Archivematica to view administrative, technical, descriptive, and preservation metadata related to AIPs and the relationships among the metadata
• Webrecorder.io: Manual capture of sites, creates WARC file

Lifecycle Environment

• Storage Pathways
•   Steps in Manage Digital workflow showing Storage Areas and Activities
•    Submission Storage -> Ingest Storage -> Archival Storage -> Dissemination Storage
• Digital Content Management Workflow
• Diagram showing steps in the Manage Digital Workflow and the Storage Areas.
• External Content Creators
• Archives Collections Security Requirements
• Archives Management Systems provide support for the archival workflow, including appraisal, accessioning, description, arrangement, publication of finding aids, collection management, and preservation.”

Digital Preservation workflows may involve the integration of several tools, such as:

• ArchivesSpace – Archivematica – ArchivesSpace
• ArchivesSpace - atom – Archivematica – atom
• BitCurator – ArchivesSpace – Archivematica - ArchivesSpace

Monitor digital files with Fixity and Binder. Look at mapping standards and tools to your local situation. Evaluate how these meet your Mission, Tasks, Components:

• What is our mission and goals? 
• Break them into tasks to accomplish
• Determine where they happen during workflow
• Identify
• Information inputs
• Digital content inputs
• Intended output
• Decide how these fit with the lifecycle storage areas
• Select tools that assist us accomplish our goals then Monitor, Adjust, Repeat

Digital Preservation Matters.

Digital Preservation Tools on Github.

Digital Preservation Tools on Github. Chris Erickson. Blog. August 2015.
     While looking for a particular tool I came across several others that look interesting. I have not yet tried them, but this is a reminder that I need to check into them. 

• epubcheck: a tool to validate EPUB files. It can detect many types of errors in EPUB. OCF container structure, OPF and OPS mark-up, and internal reference consistency are checked. EpubCheck can be run as a standalone command-line tool or used as a Java library.

• preservation-tools: Bundles a number of preservation tools for all file types and tools in a modular way. Includes:
• PdfHeaderChecker (able to detect the software used to create a PDF),
• PdfAValidator (Checks via PDFBox if a PDF/A is valid. Runs through a folder and picks out only PDF/A-files),
• iTextRepairPdf (take a PDF-file and copies the content page-per-page to a new, PDFA1-conform PDF-file)
• PdfToImageConverter (Converts PDF Files in a certain folder to JPEGs page-per-page)
• PdfTwinTest (compares the two PDF line-by-line and puts out differences. This is handy for after-Migration Quality-Checking)

• wail: Web Archiving Integration Layer (WAIL). A graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages.

• db-preservation-toolkit. The Database Preservation Toolkit allows conversion between Database formats, including connection to live systems, for purposes of digitally preserving databases. The toolkit allows conversion of live or backed-up databases into preservation formats such as DBML or SIARD, XML-based formats created for the purpose of database preservation. The toolkit also allows conversion of the preservation formats back into live systems to allow the full functionality of databases. For example, it supports a specialized export into MySQL, optimized for PhpMyAdmin, so the database can be fully experimented using a web interface.

• DPFManager. DPF Manager is an open source modular TIFF conformance checker that is extremely easy to use, to integrate with existing and new projects, and to deploy in a multitude of different scenarios. It is designed to help archivists and digital content producers ensure that TIFF files are fit for long term preservation, and is able to automatically suggest improvements and correct preservation issues. The team developing it has decades of experience working with image formats and digital preservation, and has leveraged the support of 60+ memory institutions to draft a new ISO standard proposal (TIFF/A) specifically designed for long term preservation of still-images. An open source community will be created and grown through the project lifetime to ensure its continuous development and success. Additional commercial services will be offered to make DPF Manager self-sustainable and increase its adoption.

• PreservationSimulation. This project is to provide baseline data for librarians and researchers about long-term survival rates of document collections. We have developed computer simulations to estimate document failure rates over a wide variety of conditions. The data from these simulations should be useful to stewards of such collections in planning and budgeting for storage and bandwidth needs to protect their collections.

• flint.  Facilitate a configurable file/format validation. Its underlying architecture is based on the idea that file/format validation almost always has a specific use-case with concrete requirements that may differ from a validation against the official industry standard of a given format. The following are the principle ideas we've implemented in order to match such requirements.

• excel. Regarding the second issue: how to best retain formulas and other essential components of spreadsheets, like Excel, one of our data curators, John McGrory (U of Minnesota), just published a tool in GitHub that can help. In our data repository, we use the tool each time a dataset is submitted and zip these resulting files as the "Archival Version of the Data." Download the software at http://z.umn.edu/exceltool. See also a description of what the tool does: http://hdl.handle.net/11299/171966

Digital Preservation Matters.

POWRR Tool Grid

POWRR Tool Grid. COPTR Consortium. November 2014.
     The Digital POWRR Project has produced version 2 of the Digital POWRR Tools Grid. The Grid, which helps practitioners find software tools to solve their digital preservation challenges, provides information about almost 400 digital preservation tools.The Tools Grid can also be found on a new domain for community owned digital preservation resources: Digipres Commons.

Digipres Commons highlights useful collaborative preservation resources from around the web as well as hosting these other collaborative services:

• The COPTR tools registry
• The Digital Preservation Question and Answer site
• The File Formats aggregation service

The main topics of tools, subdivided by material or format, are:

•  Access, Use and Reuse 
•  Create or Receive (Acquire) 
•  Cross-Lifecycle Functions 
•  Dispose 
•  Ingest 
•  Preservation Action 
•  Preservation Planning 
•  Store 

Digital Preservation Matters.

File identification ...let's talk about the workflows

File identification ...let's talk about the workflows. Jenny Mitcham. Digital Archiving at the University of York. 27 November 2015.
     When adding files to a digital archive, an important questions is "What file formats have we got here?" Knowing this can:

• determine the right software to open the file and view the contents 
• start the conversation with the data provider about what formats are best to use for archiving
• discuss the risks on the format and define a migration pathway for preservation and/or access

There are many tools for working with formats; each tool has strengths and weaknesses. Defining a workflow can help determine how best to use these tools, how to interact with them, or if manual steps should be taken instead. File identification tools are often incorporated into digital preservation systems that may determine the workflow in using the tools. Additional workflow questions around format tools include:

• what should happen if ingested data can't be identified?  
• should the curator/digital archivist be able to over-ride file identifications?
• what should happen if there is more than one possible identification for a file?
• is there a sustainable manual identification process if tools cannot identify a file? 
• how to contribute to file format registries such as PRONOM
• is the digital preservation system configurable enough to resolve these questions? 

Their Archivematica development work is focusing in the first instance on allowing the digital curator to see a report of the files that are not identified in order to understand the problem.

[Our Rosetta system has a format library that handles these questions, as well as a user driven Format Working Group that helps resolve questions and interacts with PRONOM if there are questions, changes or new additions. - Chris]

Digital Preservation Matters.

A Selection of Research Data Management Tools Throughout the Data Lifecycle

A Selection of Research Data Management Tools Throughout the Data Lifecycle. Jan Krause. Ecole polytechnique fédérale de Lausanne. September 9, 2015. [PDF]
     This article looks at the data lifecycle management phases and the many tools that exist to help manage data throughout the process. These tools will help researchers make the most out of their data, save time in the long run, promote reproducible research, and minimize the risks with the data. The lifecycle management phases are: discovery, acquisition, analysis, collaboration, writing, publication and deposit in trusted data repositories.  There are tools in each of the areas. A few of the many tools listed are:

• How to Develop a Data Management and Sharing Plan
• DMPonline and DMPTool  to help write data management plans.
• Recommended Data Formats
• ownCloud synchronizes and shares data on several computers 
• re3data global registry of 1,200+ research data repositories in different disciplines

It is important to use appropriated data and metadata standards, especially data formats, which should happen at the beginning since these are difficult to after the project is started.

Digital Preservation Matters.

File identification tools, part 5: FITS

File identification tools, part 5: FITS. Gary McGath. File Formats Blog.  June 25, 2015.
The File Information Tool Set (FITS), which aggregates results from several file identification tools, was created by the Harvard University Libraries and is available in Github. FITS uses Apache Tika, DROID, ExifTool, FFIdent, JHOVE, the National Library of New Zealand Metadata Extractor, and four Harvard tools.  The tool can be used in the ingest process; it processes directories and subdirectories, and produces a single XML output file in various schemas. It can be run as a standalone tool or incorporated with other tools, and can be configured to determine which tools to run and which extensions to examine.  Documentation is found on Harvard’s website.

Digital Preservation Matters.

Benchmarks for Digital Preservation tools

Benchmarks for Digital Preservation tools. Kresimir Duretec, et al. Vienna University of Technology and University of Toronto. October 2015.
     "Creation and improvement of tools for digital preservation is a di cult task without an established way to assess any progress in their quality." Software benchmarking is used to "provide objective evidence about the quality of software tools" but the digital preservation field is "still missing a proper adoption of that method." This paper looks at benchmarks and proposes a model for digital preservation and a standardized way to "objectively compare various software tools relevant to the digital preservation community."

Digital Preservation Matters.

DPOE Interview with Danielle Spalenka of the Digital POWRR Project

DPOE Interview with Danielle Spalenka of the Digital POWRR Project. Susan Manus, Barrie Howard. The Signal. July 20, 2015.
     Article about an interview with Danielle Spalenka, Project Director for the Digital POWRR Project. They had a National Leadership Grant to investigate digital preservation at institutions with limited resources. They have prepared a workshop, a white paper and the Tool Grid. The workshop, free through the end of 2016 with funding is from the NEH, looks at best practices and standards. 

Our review of the landscape of digital preservation instruction was that it is largely aimed at an audience beginning to come to grips with the idea that digital objects are subject to loss if we don’t actively care for them. There are lots of offerings discussing the theory of digital preservation – the “why” of the problem – and we found that there were limited opportunities to learn the “how” of digital preservation, both on the advocacy and technical sides. We also found that other great offerings, like the Digital Preservation Management Workshop Series based at MIT, had a tuition fee that was unaffordable for many prospective attendees, especially from under-funded institutions. Our goal in this phase is to make the workshops free to attend.

"A major goal of the workshop is to discuss specific tools and provide a hands-on portion so that participants could try a tool that they could apply directly at their own institutions." It provides an  overview of how digital preservation services and tools actually relate to the standards, how to use them in a workflow, and how to advocate for implementation. The POWRR Tool Grid is now maintained by COPTR (Community Owned digital Preservation Tool Registry).

Some recommendations for those just starting out:

• First consider what type of tool you might be interested in (processing, storage, etc.) Looking at the specific function of a tool might be a good place to understand the wide variety of tools better.
• A number of tools and services offer free webinars and information sessions to learn more about a specific tool. Download the tools to gain some hands-on experience.
• Remember that digital preservation is an incremental process, and there are small steps you can take now to start digital preservation activities at your own institution. 
• Remember you are not alone! 
• See what others are doing and talking about. 
• Ask a question on the Digital Preservation Q&A forum. You can also learn about the latest in digital preservation activities through blogs like The Signal and the blog Digital Preservation Matters. Attend a free POWRR workshop!

Related posts:

• From Theory to Action: Good Enough Digital Preservation for Under-Resourced Cultural Heritage Institutions
• Phase Two of POWRR: Extending the Reach of Digital Preservation Workshops
• POWRR Tool Grid
• Digital Preservation Tool Grid.

Digital Preservation Matters.