Introduction to Free and/or Open Source Tools for Digital Preservation

Introduction to Free and/or Open Source Tools for Digital Preservation. Max Eckard. University of Michigan. May 16, 2016.
     The post refers to a workshop that was given as part of the Personal Digital Archiving 2016 conference, entitled "Introduction to Free and/or Open Source Tools for Digital Preservation". This workshop introduced participants to a mix of open source and/or free software to review personal digital archives and "perform preservation actions on content to ensure its long-term authenticity, integrity, accessibility, and security". The presentation slides and google doc file are available and contain all the links and additional information.

The table of contents:

Pt. 1: Characterizing Content

WinDirStat

DROID (Digital Record Object IDentification)

bulk_extractor

Pt. 2: Migrating Files to Preservation Formats

      Still Images

IrfanView

ImageMagick (Demo)

      Text(ual) Content

Adobe Acrobat Pro

Ghostscript

      Audio and Video

FFmpeg

HandBrake

Pt. 3: Metadata for Digital Preservation

Descriptive Metadata

Microsoft Word

Adobe Acrobat

Technical Metadata

ExifTool

Preservation Metadata

      MD5summer

Digital Preservation Matters.

US nuclear force still uses floppy disks

US nuclear force still uses floppy disks. BBC News Services. 26 May 2016.
     A government report shows that the US nuclear weapons forces ( intercontinental ballistic missiles, nuclear bombers and tanker support aircraft) still use a 1970s-era computer system and 8-inch floppy disks. The GAO said there were "legacy systems" which need to be replaced. Legacy systems cost about $61bn a year to maintain. "This system remains in use because, in short, it still works."  "However, to address obsolescence concerns, the floppy drives are scheduled to be replaced with secure digital devices by the end of 2017." According to the report, the US treasury systems, still use a system written in "assembly language code".

Digital Preservation Matters.

List of analog media inspection templates/forms

List of analog media inspection templates/forms. Katherine Nagels, et al. May 6, 2016.
     This  is a list of freely available analog media inspection templates, forms, or reports. Anyone is free to add contributions.  Appropriate additions may include:

• inspection reports/forms/templates
• condition reports/forms/templates
• instructional guides for inspecting or assessing the condition of analog film, audio, or video

TEMPLATES/FORMS

• Guggenheim Museum, Conservation Department (date unknown): Media Report: Tape; Media Report: Film
• Matters in Media Art (2015): Condition Reports: video, audio, slides, film, computer-based artwork, display hardware
• Tanisha Jones, Moving Image Archiving & Preservation, New York University (2005): Video Prioritization, Selection, and Inspection Overview and Form
• Independent Media Arts Preservation (IMAP) (c.2006): Videotape Inspection Form
• National Film and Sound Archive Australia (c.2008): Condition Report Form for Film
• Filmoteca Española (1996): Informe de inspección de materiales (película)
• National Film Preservation Foundation (NFPF) (2004): Print Condition Report for Film
• Columbia University Libraries (2008): Audio and Moving Image Survey Tool
• University of Washington Libraries: Film Condition Report; Videotape/Audiotape Condition Report

IDENTIFICATION RESOURCES

• University of Illinois at Urbana-Champaign Preservation Self-Assessment Program: Format ID Guide
• California Audiovisual Preservation Project (2013): Audiovisual Formats: A Guide to Identification
• Texas Commission on the Arts (2004): Videotape Identification and Assessment Guide
• Indiana University Field Audio Collection Evaluation Tool (FACET) (2007): Format Characteristics and Preservation Problems
• Video Preservation Web Site (2007): Video Format Identification Guide

Digital Preservation Matters.

The Governance of Long-Term Digital Information

The Governance of Long-Term Digital Information. IGI 2016 Benchmark. Information Governance Initiative. May 18, 2016. [PDF]
     “The critical role of digital . . .archives in ensuring the future accessibility of information with enduring value has taken a back seat to enhancing access to current and actively used materials. As a consequence, digital preservation remains largely experimental and replete with the risks . . . representing a time bomb that threatens the long-term viability of [digital archives].”

1. We have a problem. Nearly every organization has digital information they want to keep for 10 or more years.
2. The problem is technological, most often a storage problem.
3. The problem is business related. It is not related to just archives, libraries or museums. 
4. The problem is a legal problem. Legal requirements are the main reason organizations keep 
digital information longer than ten years
5. We know what we must do, but are we doing it? In a survey 97 percent said they are aware that digital information is at risk of obsolescence but three fourths are just thinking about it or have no strategy. Only 16% have a standards-based digital preservation system.

• “Most records today are born digital."
• Digital assets should be considered business-critical information and steps taken to keep them usable long into the future
• Most organizations are not storing their long-term digital assets in a manner sufficient to ensure their long-term protection and accessibility.
   

How are they being kept? According to a survey:

• Shared Network Drive                                68%
• Business Applications (e.g. CRM, ERP)        52%
• Enterprise Content Management System     47%
• Disk or Tape Backup Systems                      44%
• Records Management System                      43%
• Application-specific Archiving (e.g. email)  33%
• Removable Media (e.g. CD or USB)              22%
• Enterprise Archiving System                       14%
• Long-term Digital Preservation System        11%
• Other                                                          9%
• Commodity Cloud Storage (e.g. Amazon)      8%
• I don't know                                                 1%

Where to start? Some recommendations:

• Triage right now the materials that are in serious danger of being lost, damaged, or rendered inaccessible.
• Conduct a formal assessment so that you can benefit from strategic planning and economies of scale.
• Address the Past, Protect the Future
• Catalog the Consequences of not being able to access and rely upon your own information
• Build Your Rules for Protection and accessibility
• Assess the IT Environment

Digital Preservation Matters.

One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats

One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats. Andy Klein. Backblaze. May 17, 2016.
     Backblaze reports statistics for the first quarter of 2016 on 61,590 operational hard drives used to store encrypted customer data in our data center. The hard drives in the data center, past and present, totaled over one billion hours in operation to date.The data in these hard drive reports has been collected since April 10, 2013. The website shows the statistical reports of the drive operations and failures every year since then. The report shows the drives (and drive models) by various manufacturers, the number in service, the time in service, and failure rates. The drives in the data center come from four manufacturers, most of them are from HGST and Seagate. Notes:

• The overall Annual Failure Rate of 1.84% is the lowest quarterly number we’ve ever seen.
• The Seagate 4TB drive leads in “hours in service” 
• The early HGST drives, especially the 2- and 3TB drives, have lasted a long time and have provided excellent service over the past several years.
• HGST has the most hours in service

Related posts:

• Backblaze releases raw data on all 41,000 HDDs in its data center
• Backblaze takes on Google, Amazon with storage at half a penny a gigabyte
• Failure rates among consumer-class hard drives
• How do you preserve your personal data forever? We ask an expert

Digital Preservation Matters.

IBM Scientists Achieve Storage Memory Breakthrough

IBM Scientists Achieve Storage Memory Breakthrough. Press release. 17 May 2016.
     IBM Research demonstrated reliably storing 3 bits of data per cell using phase-change memory. This technology doesn't lose data when powered off and can endure at least 10 million write cycles, compared to 3,000 write cycles for an average flash USB stick. This provides "fast and easy storage" to capture the exponential growth of data.

Digital Preservation Matters.

Floppy Disk Format Identifer Tool

Floppy Disk Format Identifer Tool. Euan Cochrane. Digital Continuity Blog. May 13, 2016.
     Euan created this tool https://github.com/euanc/DiskFormatID (which he documents in this great blog post) to:

1.     “Automatically” identify floppy disk formats from kryoflux stream files.
2.     Enable “simple” disk imaging workflows that don’t include a disk format identification step during the data capture process.

The tool processes copies of floppy disk data saved in the kryoflux stream file format, creates a set of disk image files formatted according to assumptions about the disk’s format, and allows the user to try mounting the image files as file systems. It requires the Kryoflux program to function. The documentation also provides detailed information on how to use it, along with other interesting information.

Digital Preservation Matters.

The Center for Jewish History Adopts Rosetta for Digital Preservation and Asset Management

The Center for Jewish History Adopts Rosetta for Digital Preservation and Asset Management. Ex Libris. Press Release. May 12, 2016.
     After a thorough search process, the Center for Jewish History selected the Ex Libris Rosetta digital asset management and preservation solution. They wanted a system to handle their comprehensive list of requirements for both long‑term digital preservation and robust management of digital assets, including the ability to interface with their other systems.

The Center’s partners are American Jewish Historical Society, American Sephardi Federation, Leo Baeck Institute, Yeshiva University Museum, and YIVO Institute for Jewish Research.  The collections include more than five miles of archival documents, over 500,000 volumes, and thousands of artworks, textiles, ritual objects, recordings, films, and photographs.

Digital Preservation Matters.

Looking Across the Digital Preservation Landscape

Looking Across the Digital Preservation Landscape. Margaret Heller. ACRL TechConnect Blog. April 25, 2016.
     "When it comes to digital preservation, everyone agrees that a little bit is better than nothing." The article cited refers to two presentations from Code4Lib 2016, “Can’t Wait for Perfect: Implementing “Good Enough” Digital Preservation” by Shira Peltzman and Alice Sara Prael, and “Digital Preservation 101, or, How to Keep Bits for Centuries” by Julie Swierczek. This article mentions two major items about digital preservation:

1. Digital preservation doesn’t have to be hard, but it does have to be intentional.
2. Digital preservation requires institutional commitment. 

Understanding all the basic issues and what your options are can be daunting. They had a committee that started examining born digital materials, but expanded the  focus to all digital materials because it made it easier to test their ideas. Some of the tasks they accomplished included: created a rough inventory of digital materials, a workflow manual, and secured networked storage  to replace all removable hard drives used for backups. "While backups aren’t exactly digital preservation, we wanted to at the very least secure the backups we did have". The inventory and workflow manual are living documents and are useful for identifying gaps in the processes.

They also looked at the end-to-end systems available for digital preservation, such as Preservica, ArchivesDirect, and Rosetta. Migrating from one system to another if you change your mind may involve some very difficult processes, so people may tend to stay with providers.  Another option is to join a preservation network, such as Digital Preservation Network (DPN) or APTrust, that have the larger preservation goal ensuring long-term access to material even if the owning institution disappears.

Sustainable Financing for many is the crux of the digital preservation problem. "It’s possible to do a sort of ok job with digital preservation for nothing or very cheap, but to ensure long term preservation requires institutional commitment for the long haul, just as any library collection requires."

Digital Preservation is receiving more attention digital preservation lately and hopefully more libraries will see this as a priority.

Digital Preservation Matters.