Design Requirements for Better Open Source Tools

OSS4Pres 2.0: Design Requirements for Better Open Source Tools. Heidi Elaine Kelly. bloggERS! April 25, 2017.
     Free and Open Source Software need to "integrate easily with digital preservation institutional systems and processes.” The FOSS Development Requirements Group created a design guide for to ensure easier adoption of open-source tools and their integration with other software and tools.

Minimum Necessary Requirements for FOSS Digital Preservation Tool Development. The premise is that "digital preservation is an operating system-agnostic field."

Necessities

• Provide publicly accessible documentation and an issue tracker
• Have a documented process so people can contribute to development, report bugs, and suggest new documentation
• Every tool should do the smallest possible task really well; if you are developing an end-to-end system, develop it in a modular way in keeping with this principle
• Follow established standards and practices for development and use of the tool
• Keep documentation up-to-date and versioned
• Follow test-driven development philosophy
• Don’t develop a tool without use cases, and stakeholders willing to validate those use cases
• Use an open and permissive software license to allow for integrations and broader use

Recommendations

• Have a mailing list or other means for community interaction
• Establish community guidelines
• Provide a well-documented mechanism for integration with other tools/systems
• Provide functionality of tool as a library, separate UI from the actual functions
• Package tool in an easy-to-use way, that supports any dependencies
• Provide examples of functionality for potential users
• Consider the long-term sustainability of the tool
• Consider a way for internationalization of the tool  

Digital Preservation Matters.

Using Open-Source Tools to Fulfill Digital Preservation Requirements

OSS4EVA: Using Open-Source Tools to Fulfill Digital Preservation Requirements. Marty Gengenbach, et al. Code4Lib. 2016-10-25.
     Open-source software has played an increasingly prominent role in digital preservation, such as LOCKSS, DSpace, and DROID. The number and variety of such tools has increased, there was a growing need among preservationists to assess how and when to adopt particular tools so that they could better support their institutions’ specific requirements and workflows.  Open-source projects allows the user community to contribute by developing and documenting tools.

There are some challenges with open source programming.

• Perceptions of instability:  One challenge is the perception that these tools are "inherently unstable and therefore present a risk". 
• Resources and funding: Administrators often are reluctant to commit resources to an open source project. Funding problems can threaten the long-term sustainability of open source tools.
• System updates: Open source tools require regular patches, updates, and upkeep. Without this, the tool would be outdated, and open to security holes. "The choice to maintain an unsupported version of a particular open-source tool simply because it meets (or has been customized to meet) an organization’s needs is problematic. For what an institution may stand to gain from this tool in terms of functionality and local integration, it may stand to lose in terms of the stability of a mainstream code release, the risk to information security, and the likelihood that the tool in question will become increasingly less functional and reliable as it ages".
• Integration. Integrating open-source tools into institutional workflows can be a challenge, taking into account software dependencies, system requirements, and local configuration to put the tools into a production environment. This can require a considerable time and resources. 

One of the possible benefits is that institutions can customize open source tools for use within a specific context, but that comes with its own hurdles, such as reducing the ability to draw on the user community.  The digital preservation open source landscape has evolved from a scattered set of standalone tools designed to complex software environments. "Nevertheless, these tools still are not watertight." There are real concerns about open-source tools that can pose serious risks to collections.

Digital Preservation Matters.

Rosetta Knowledge Center

Rosetta Knowledge Center. Ex Libris. April 17, 2017.
     One of the things that I like about Rosetta, is the Ex Libris commitment to an open system. While the software may be proprietary, the essential content is open. The permanent objects and metadata are stored openly, so that they can be accessed or managed outside of the Rosetta software.

Another area that Ex Libris has opened is their Knowledge Center. This is very helpful in training new employees, learning new things about the software, or refreshing my memory. The open website includes:

• Product Documentation
• Training: Learn new skills with tutorials, recorded training and other materials
• Release Notes about the features and capabilities of each product version
• Implementation Guides that explain the methodology and requirements
• Knowledge Articles providing answers to help answer questions

Digital Preservation Matters.

iPRES workshop report: Using Open-Source Tools to Fulfill Digital Preservation Requirements

iPRES workshop report: Using Open-Source Tools to Fulfill Digital Preservation Requirements.  Jenny Mitcham. Digital Archiving at the University of York. 12 November 2015.
     The ‘Using Open-Source Tools to Fulfill Digital Preservation Requirements’ workshop provided a place to talk about open-source software and share experiences about implementing open-source solutions. Archivematica, Archivespace, Islandora and BitCurator (and BitCurator Access) were also discussed.

Sam Meister of the Educopia Institute talked about a project proposal called OSSArcFlow. "This project will attempt to help institutions combine open source tools in order to meet their institutional needs. It will look at issues such as how systems can be combined and how integration and hand-offs (such as transfer of metadata) can be successfully established". The lessons learned (including workflow models, guidance and training) will be available to others besides the 11 partners. 

Digital Preservation Matters.

Taking Control: Identifying Motivations for Migrating Library Digital Asset Management Systems

Taking Control: Identifying Motivations for Migrating Library Digital Asset Management Systems. Ayla Stein, Santi Thompson. D-Lib Magazine. September/October 2015.
     "Digital asset management systems (DAMS) have become important tools for collecting, preserving, and disseminating digitized and born digital content to library patrons." This article looks at why institutions are migrating to other systems and in what direction. Often migrations happen as libraries refine their needs. The literature on the migration process and the implications is limited; this article provide several case studies of repository migration.A presentation by Lisa Gregory "demonstrated the important role digital preservation plays in deciding to migrate from one DAMS to another and reiterated the need for preservation issues and standards to be incorporated into the tools and best practices used by librarians when implementing a DAMS migration".  Repository migration gives institutions the opportunity to move from one type of repository, such as home grown or proprietary, to another type.  Some of the reasons that institutions migrated to other repositories (by those ranked number 1) are:

• Implementation & Day-to-Day Costs
• Preservation
• Extensibility
• Content Management
• Metadata Standards

Formats they wanted in the new system included:

Response	Num.	%
PDF	28	98
JPEG	26	90
MP3	22	76
JPEG2000	21	72
TIFF	21	72
MP4	19	66
MOV	17	59
CSV	16	55
DOC	13	45
DOCX	12	41

For metadata, they wanted the new system to support multiple metadata schema; administrative, preservation, structural, and/or technical metadata standards; local and user created metadata, and linked data. In addition, METS and PREMIS were highly desirable.

The new system should support, among others:

• RDF/XML
• Ability to create modules/plugins/widgets/APIs, etc.  
• Support DOIs and ORCIDs

Preservation features and functionality were the ability to:

• generate checksum values for ingested digital assets.
• perform fixity verification for ingested digital assets.
• assign unique identifiers for each AIP
• support PREMIS or local preservation metadata schema.
• produce AIPs.
• integrate with other digital preservation tools.
• synchronize content with other storage systems (including off site locations).
• support multiple copies of the repository — including dark and light (open and closed) instances.

The survey suggests that "many information professionals are focused on creating a mechanism to ensure the integrity of digital objects." Other curatorial actions were viewed as important, but some "inconclusive results lend further support claims of a disconnect between digital preservation theory and daily practices". About two-thirds were moving to open source repositories, while one fifth were moving to proprietary.

Digital Preservation Matters.

Viewshare: Interfaces to our heritage

Viewshare: Interfaces to our heritage. Library of Congress. Website. April 2013.
Viewshare, an open source instance of Recollection, is a free platform for generating and customizing views (interactive maps, timelines, facets, tag clouds) that allow users to experience your digital collections.
Functionality:

1. Ingest collections from spreadsheets or MODS records.
2. Generate distinct interactive visual interfaces to digital collections, including maps and timelines, and sophisticated faceted navigation.
3. Just copy-paste to embed interface in any webpage. Provide users with new ways to explore content
   
   
   Digital Preservation Matters.

Dream of perpetual access comes true!

Dream of perpetual access comes true! Jeffrey van der Hoeven. Open Planets Foundation.  29 March 2012.
The KEEP project released its final version of the open source Emulation Framework software. This project has brought emulation in the digital preservation context to the next level, that is, user friendly.  The easy to install package runs on all major computer platforms.  It automates several steps:

1. identify what kind of digital file you want to render;
2. find the required software and computer platform you need;
3. match the requirements with available software and emulators;
4. install the emulator;
5. configure the emulator and prepare software environment;
6. inject the digital file you selected into the emulated environment;
7. give you control over the emulated environment.

 The software supports six different computer platforms out of the box: x86, Commodore 64, Amiga, BBC Micro, Amstrad, Thomson, by using seven open source emulators which are distributed  with the Emulation Framework. 

Digital Preservation Matters.

Free Tools for Your Preservation Toolbelt.

Free Tools for Your Preservation Toolbelt. Randy Stern, Spencer McEwen.  Harvard.  June 2011. A presentation delivered at the Open Repositories 2011 conference. 
The Digital Repository Service models “objects” rather than files.  Examples:
- Delivery, archival master, and production master images comprise one object
- All images and OCR text for a book comprise one object

Object Preservation Metadata: Digital preservation requires accurate and sufficient technical metadata to support preservation planning and activities. Descriptive metadata is also valuable for identification and management by curators.  Standards-based schema maximize tool support and ability to exchange data with other repositories.

Here are some tools they use, which are open source or will be soon.

Tool 1 - FITS (File Information Tool Set)
Identifies, validates, and extracts technical metadata from files

Tool 2 - OTS-Schemas (Object Tool Set Schemas)
Java library for reading and writing documents in common XML schemas

Tool 3 - OTS (Object Tool Set)
Java library for creating, reading, updating, and writing METS Object Descriptors

Tool 4 - BatchBuilder
Builds OTS METS objects (and SIP) from directory hierarchies of content files

Library of Congress Digital Preservation Newsletter.

Library of Congress Digital Preservation Newsletter. Library of Congress.  August 2011. [PDF]

The newsletter includes information on:

•  “Make it Work: Improvisations on the Stewardship of Digital Information,” 
• All About Archiving the Web 
• Possible uniform law on the authentication of online legal materials
• Exploring Cultural Heritage Collections With Recollection
• Recollection is a free and open source platform that lets archivists, librarians, scholars and curators create easy to navigate web interfaces (like maps, timelines, facets, tag clouds) to their digital, cultural heritage collections.
• Finding digital preservation training.  The training calendar.
• Digital Time Capsules and our "Digital Afterlife"
• Creating and organizing personal digital content for future access.
• The Signal: Library of Congress blog to discuss digital stewardship in a way that is informative and appealing.
• Tending the machines
• What skills does a digital archivist or librarian need?  Skills students need to compete in the archives and libraries job market.  Expertise with programming, formats and standards is, of course, very important.  But other talents have a greater bearing on success in today’s workplace. Such as:
• an ability to understand and adapt to new ways of using technology
• eagerness to help refine how things are done
• a basic understanding of how the different system parts contribute to doing the job at hand
• ability to bridge two distinct social camps: the highly technical and the highly not-technical
• how choose among these tools and software options to meet the needs of users
• communication skills, including presentation, writing, speaking and persuading
• ability to social media and to integrate photographs, graphics and video with text to get the right message out to as many people as possible

Digital Preservation Matters - 30 October 2009

Copyright and Cultural Institutions: Guidelines for U.S. Libraries, Archives, and Museums. Peter B. Hirtle, et al. October 2009. [Book and 275p. PDF]

This book can help libraries learn how they can use the internet to provide access to their collections and comply with copyright laws. The book is also available in PDF form on the website. It “addresses the basics of copyright law and the exclusive rights of the copyright owner, the major exemptions used by cultural heritage institutions, and stresses the importance of “risk assessment” when conducting any digitization project.” A section on ‘Digital preservation and replacement copies’ is important to read to learn more about what we can do and can’t do.

---

Universities offering new perks to broke students. Carol Warner. HigherEdMorning. October 25, 2009.

An initiative in Florida makes more than 120 textbooks available to students to download for free. It also sells the books at a discount because findings show:

• 22% of students are “uncomfortable” reading from a computer screen

• 75% of students prefer to read print copy,

• 60% of students would buy a discounted book even if the textbook was available for free online.

---

Microsoft opens Outlook format, gives programs access to mail, calendar, contacts. John Fontana. HigherEdMorning. October 27, 2009.

Microsoft announced it will provide patent- and license-free use rights to the format behind its Outlook Personal Folders. It will document and publish the .pst format, which is used for the email, calendar, and contact functions. This will explain how to parse the contents of the file and how to access that data from other software applications.

---

iTunes for college courses? It’s true. Carin Ford. HigherEdMorning. October 24, 2009.

The University of Virginia now offers over 1,000 lectures, videos, etc. as a free digital download from iTunes U. Others have done this for a long time. But they have combined other features. An interesting feature is that if a student subscribes to a specific course, new material will be downloaded automatically to his iTunes library.

---

Obama Drupal-ing around; whitehouse.gov goes open source. Richi Jennings. Computerworld. October 26, 2009.

White House has chosen Drupal, an open-source content management software, to run the whitehouse.gov Web site.

---

High Volume Document Storage Creates New Headaches for Content Managers. Steve Jones. CMS Wire. Oct 26, 2009.

Digital storage centers are filling rapidly. Some look at ‘single-instance storage’ as a viable option for organizations trying to reduce digital storage requirements. Single instancing is based on the principle of keeping one copy of a digital file that multiple users share and eliminating duplication.

---

Open Access to Research Is Inevitable, Libraries Are Told. Jennifer Howard. The Chronicle of Higher Education. October 15, 2009.

A panel told ARL libraries that public access to research is "inevitable," but it will take work to get there. Faculty are starting to understand that open access to research has to happen in order to have the most scholarly opportunities. The US is far behind other countries regarding access to research. Researchers who don’t have the latest research can't fully participate in the academic discussions. The National Science Foundation plans to build an international, large-scale data-curation network.