The File Discovery Tool - A simple tool to gather file and filepath information, and ingest into our Rosetta Digital Archive

The File Discovery Tool. Chris Erickson. Brigham Young University. November 29, 2018.
     We have created a File Discovery Tool that analyzes directories of objects and prepares a spreadsheet of all the files it discovers for preservation/ingest. This file allows the curators to discover and work with the materials, select those that need to be preserved, and then add collection and other metadata information. The tool fits our workflow, but the source code may be useful for others trying to accomplish a similar task.

A sample command to run the tool:

>> java -jar FileDiscovery.jar [path name of files to check] [output path name for saving the report]
>> java -jar C:\FileDiscovery\FileDiscovery.jar "R:\test\objects"  C:\output\files

 The commands and syntax are outlined in a brief document: File Discovery Outline

  

The spreadsheet that is created has the following column headings:

 FILENAME, ITEM ID, FILEPATH, BYTESIZE, SIZE, COLLECTION, IE_LEVEL, DATE_CREATED, DATE_MODIFIED, TITLE, CREATOR, DESCRIPTION, RIGHTS_POLICY

Metadata can be added as needed before ingesting the content into Rosetta.

The files and the metadata can then be submitted to Rosetta using the csv option in the Rosetta File Harvester tool by adding in a second row of Dublin Core names in order to map the column. A standard template has been created to help in preparing the file for ingest and is found on the resources page: RosettaFile Ingest template for Excel, or (PDF)

The source is available at https://bitbucket.org/byuhbll/filediscovery

Digital Preservation Matters.

The File Harvester tool - Our tool for ingesting content to our Rosetta Digital Archive

The File Harvester tool. Chris Erickson. Brigham Young University. November 29, 2018. 

     We have created a harvester tool for harvesting, processing, and submitting content to Rosetta. Our Library IT department has made this open source. The tool fits our workflow, but the source code may be useful for others trying to accomplish a similar task.

The File Harvester tool gathers content from several different sources:

• Our hosted CONTENTdm (cdm)
• Open Journal System (ojs)
• Internet Archive (ia)
•  Unstructured files in a folder with metadata in a spreadsheet (csv)

The tool creates SIPs by adding objects and metadata from the specified source, by creating a Rosetta mets xml file and a Dublin core xml file; and by putting it in the structure for our Rosetta system. The objects can either be on the hosted system or in a source folder. The harvest tool can also submit the content to Rosetta for ingest.

The structure is:

1. Folder: collection-itemid and it contains the dc.xml and subfolder content 
2. Sub-Folder: content and it contains the mets.xml and the folder streams 
3. Sub-Folder: streams which contains the file objects

The commands and syntax are outlined in a brief document on the Resources page:
RosettaFile Harvester outline

The source is available at: https://bitbucket.org/byuhbll/rosetta-tools

Digital Preservation Matters.

Integrating Research Data management and digital preservation systems at the University of Sheffield

Integrating Research Data management and digital preservation systems at the University of Sheffield. Chris Loftus. Digital Preservation Coalition. 31 May 2017.
     The University Library is leading the active management and curation of research data within the institution. This includes implementing a research data catalogue and repository powered by Figshare. They safeguard library collections and University assets of the University using Rosetta, a digital preservation platform from Ex Libris. "We are now working with figshare and Ex Libris to integrate both services to provide seamless preservation of published research data across the research lifecycle." Which will

• provide a complete lifecycle data management service for the university’s research community; 
• identify, understand and act on risks associated with preserving data sets; 
• better inform advice and guidance around use of data formats for sharing and preservation purposes; and 
• encourage researchers to share their data more openly with others by guaranteeing the long term sustainability of that data.

Initial integration work uses the OAI-PMH protocol and METS packages to transfer content efficiently. Rosetta will be the dark archive, with figshare the interface for researchers and external users.

File formats issues: Research data is often in niche and proprietary formats. Of the material currently deposited in the archive, only a small percentage was recognised by a Droid survey. They will need to invest some time to identify and plan for these formats, and hopefully the work will be of use to the wider digital preservation community.

Metadata: They plan to improve the quality and volume of metadata accompanying research data. Material from researchers often lacks needed metadata, which can cause future data access issues. They are investigating solutions.

Digital Preservation Matters.

Ex Libris joins the Open Preservation Foundation

Ex Libris joins the Open Preservation Foundation. Becky McGuinness. Press Release. Open Preservation Foundation. June 1, 2017.
     The Open Preservation Foundation announced that Ex Libris is its newest charter member. "Ex Libris’ Rosetta is an end-to-end digital asset management and preservation solution for libraries, archives, museums and other institutions, enabling institutions to safely and securely collect, manage, publish, deliver, and ensure longevity for digital information of many different types. With Rosetta’s unique content preservation planning module and its Format Library knowledge base, shared by the entire Rosetta community, institutions can identify format risks, evaluate mitigation alternatives, and select the best preservation actions."  "Rosetta reflects Ex Libris involvement in industry standards and commitment to extensibility and open architecture."  "Rosetta itself is based on an open architecture that allows customers to easily use Rosetta with external tools and plugins such as JHOVE and other open-source software. By supporting OPF, we can further improve open-source tools for the benefit of all."
 

Digital Preservation Matters.

Rosetta Knowledge Center

Rosetta Knowledge Center. Ex Libris. April 17, 2017.
     One of the things that I like about Rosetta, is the Ex Libris commitment to an open system. While the software may be proprietary, the essential content is open. The permanent objects and metadata are stored openly, so that they can be accessed or managed outside of the Rosetta software.

Another area that Ex Libris has opened is their Knowledge Center. This is very helpful in training new employees, learning new things about the software, or refreshing my memory. The open website includes:

• Product Documentation
• Training: Learn new skills with tutorials, recorded training and other materials
• Release Notes about the features and capabilities of each product version
• Implementation Guides that explain the methodology and requirements
• Knowledge Articles providing answers to help answer questions

Digital Preservation Matters.

German hbz Consortium Selects Ex Libris Rosetta Digital Asset Management and Preservation Solution

German hbz Consortium Selects Ex Libris  Rosetta Digital Asset Management and Preservation Solution. Press Release. ProQuest. 29 November 2016.
     Hochschulbibliothekszentrum des Landes Nordrhein‑Westfalen has chosen the Ex Libris Rosetta digital asset management and preservation solution. There are more than 40 member institutions that will be able to deposit digital collections in the central Rosetta system. “Our preservation and management plans across the entire North-Rhine Westphalia region include both artifacts and modern research output. With Rosetta, we will be able preserve a wide range of data and manage digital assets on both the consortium and institutional level. Rosetta meets our current and long-term needs.” 

Digital Preservation Matters.

Victoria University of Wellington Selects Ex Libris Rosetta for Preserving and Managing Digital Assets

Victoria University of Wellington Selects Ex Libris Rosetta for Preserving and Managing Digital Assets. Press release. August 2016.
     "Victoria University of Wellington has selected Rosetta as its digital preservation and asset management solution. The Victoria University Library serves as the custodian of over 15,000 digitized historical cultural works, part of the New Zealand Electronic Text Collection, and over 11,000 born-digital theses and research projects in institutional repositories, including several other smaller digital collections. Rosetta will be a key element of the Library’s digital assets management and preservation processes and will enable researchers in any location to read or view the digital objects in the Library's extensive collections."

"Adopting Rosetta will enable us to manage, maintain, and preserve these collections in the long term, as well as grow them in the future. As our collections increase, standards of digital preservation and description become more vital to the continuity and discovery of materials for future knowledge creation. It’s not just about the students and researchers of today. It’s about the students and researchers of the future, too.”

Digital Preservation Matters.

The Center for Jewish History Adopts Rosetta for Digital Preservation and Asset Management

The Center for Jewish History Adopts Rosetta for Digital Preservation and Asset Management. Ex Libris. Press Release. May 12, 2016.
     After a thorough search process, the Center for Jewish History selected the Ex Libris Rosetta digital asset management and preservation solution. They wanted a system to handle their comprehensive list of requirements for both long‑term digital preservation and robust management of digital assets, including the ability to interface with their other systems.

The Center’s partners are American Jewish Historical Society, American Sephardi Federation, Leo Baeck Institute, Yeshiva University Museum, and YIVO Institute for Jewish Research.  The collections include more than five miles of archival documents, over 500,000 volumes, and thousands of artworks, textiles, ritual objects, recordings, films, and photographs.

Digital Preservation Matters.

File identification ...let's talk about the workflows

File identification ...let's talk about the workflows. Jenny Mitcham. Digital Archiving at the University of York. 27 November 2015.
     When adding files to a digital archive, an important questions is "What file formats have we got here?" Knowing this can:

• determine the right software to open the file and view the contents 
• start the conversation with the data provider about what formats are best to use for archiving
• discuss the risks on the format and define a migration pathway for preservation and/or access

There are many tools for working with formats; each tool has strengths and weaknesses. Defining a workflow can help determine how best to use these tools, how to interact with them, or if manual steps should be taken instead. File identification tools are often incorporated into digital preservation systems that may determine the workflow in using the tools. Additional workflow questions around format tools include:

• what should happen if ingested data can't be identified?  
• should the curator/digital archivist be able to over-ride file identifications?
• what should happen if there is more than one possible identification for a file?
• is there a sustainable manual identification process if tools cannot identify a file? 
• how to contribute to file format registries such as PRONOM
• is the digital preservation system configurable enough to resolve these questions? 

Their Archivematica development work is focusing in the first instance on allowing the digital curator to see a report of the files that are not identified in order to understand the problem.

[Our Rosetta system has a format library that handles these questions, as well as a user driven Format Working Group that helps resolve questions and interacts with PRONOM if there are questions, changes or new additions. - Chris]

Digital Preservation Matters.

Archives and SharePoint

Archives and SharePoint. Heather Emily Roberts. HerArchivist. March 8, 2016.
     Post that looks at "Is SharePoint (or other flexible cloud-based ERMS software) suitable for digital repositories of archives?"  Some pros and cons of using SharePoint as a digital repository:
Pros:

• Can lock documents against editing
• Tells you when documents were last accessed and by whom

Cons:

• Will not serve long-term needs of accessibility or use of records
• Will not support migration requirements of archival records
• Will not guarantee integrity of archival records during software updates
• Does not conform to OAIS model
• Archive preservation practices are not standard

The main recommendation if using SharePoint as an Electronic Record Management System is to export archival documents to an OAIS compliant system (some are listed).
[We use our harvest tool to import permanent SharePoint records into our Rosetta system - Chris.]

Related posts:

• New Sources and Storage Options For Rosetta
• OAI-PMH harvesting from SharePoint

 

Digital Preservation Matters.

Wowza as a Streaming Media Service at the National Library of New Zealand

Wowza as a Streaming Media Service at the National Library of New Zealand. Mathachan Kulathinal. Rosetta Tech Blog. March 2, 2016.
     The National Library of New Zealand uses the Wowza Streaming Engine Server to stream derivative copies of video and audio content stored within their Rosetta preservation system. Details of our integration with Rosetta are outlined below, configuration or environment specific details used in the implementation of this model at the Library have not been included.

According to the diagrams, the Wowza Streaming Engine service is separate from the Rosetta servers; the streaming media servers have been kept separate and independent of the Rosetta services. The servers are also used as proxy servers to re-direct http traffic as required. The post includes the implementation details.

Digital Preservation Matters.

OhioLINK Adopts Ex Libris Rosetta for Digital Preservation

OhioLINK Adopts Ex Libris Rosetta for Digital Preservation. Ex Libris. Press release. December 21, 2015.
     OhioLINK has selected the Ex Libris Rosetta digital management and preservation solution for 120 academic libraries plus the State Library of Ohio. Rosetta will ensure long-term access to the OhioLINK Electronic Journal Collection (EJC), Electronic Book Collection (EBC), Electronic Theses and Dissertations (ETD) Center, and Digital Resource Commons (DRC) collections. - OhioLINK sought a preservation system based on the Open Archival Information System (OAIS) reference model that could integrate with its existing content management systems and support a wide range of processing workflows. As a large and complex consortium, OhioLINK required a solution that could be implemented and maintained in a way that suits a wide variety of content.

Digital Preservation Matters.

Digital Curation Coordinator, OhioLINK

Digital Curation Coordinator, OhioLINK. Online posting. December 3, 2015.
     The Digital Curation Coordinator will manage OhioLINK’s implementation of the Rosetta digital preservation platform; assist in developing required policies and procedures related to digital collections and preservation; represent OhioLINK at digital curation events; and interact with OhioLINK members on issues related to digital curation. 

Digital Preservation Matters.

The Shanghai Library Selects Ex Libris Rosetta

The Shanghai Library Selects Ex Libris Rosetta. Press release. Ex Libris. November 2, 2015.
     The Shanghai Library, the second largest library in China and one of the world’s largest public libraries, chose Rosetta to manage and preserve its vast collection of digitized records such as ancient books, sound recordings, manuscripts, genealogy resources, archives (such as the Sheng Xuanhuai Archives, books and journals published in the Republic period, and the North China Daily News). Rosetta’s support for multiple languages and its customized Chinese interface will enable library staff to deposit diverse content into the system and expose a wide range of rich Chinese heritage to the world. "Rosetta was the only solution on the market that supports the whole spectrum of digital asset management and preservation, from ingest and export, to collection management and publishing.”

 

Digital Preservation Matters.

University of Alabama at Birmingham Selects Ex Libris Rosetta for Digital Preservation

The University of Alabama at Birmingham Opts for Ex Libris Alma, Primo, and Rosetta Solutions. News Release. Ex Libris. October 21, 2015.
     The University of Alabama at Birmingham (UAB) has selected the Ex Libris products, including the Rosetta digital asset management and preservation solution. "Rosetta’s end-to-end digital asset infrastructure will preserve digital resources at both libraries and keep such resources accessible for future generations.... We are acquiring Rosetta to support preservation for UAB’s digital assets, ranging from institutional memory to research data."

Digital Preservation Matters.

ProQuest to Acquire Ex Libris

ProQuest to Acquire Ex Libris. Press release. ProQuest. October 6, 2015.
ProQuest has signed an agreement to acquire Ex Libris. "The ProQuest acquisition of Ex Libris will align print and digital resources, pioneering discovery and library management solutions, and the deep content relationships that underlie our leading metadata repositories. Our current solutions will continue to be available, with ongoing enhancements and improvements. Perhaps most exciting, we will build new and innovative solutions that address customers’ evolving needs.

The products and services you know and rely upon will be enriched by the capabilities and skills of both companies. For example, imagine the value to libraries of having unified workflows for acquisition, cataloguing and circulation or a single system that deftly manages both print and electronic resources. More content choices will be available to libraries and users around the world, and we will better meet the needs of customers in their regions, in their languages and in their time zones.

ProQuest remains committed to the core principles of choice, openness and collaboration. Indeed, these are principles shared by Ex Libris.

We will continue to work with our incredible user communities for better outcomes. Both ProQuest and Ex Libris are deeply engaged with the library community and customers. Both companies benefit from active, collaborative and forward-thinking global customer bases and user groups. Learning from you, our customers and partners, has been – and will continue to be – extremely important to us. We will be looking to you for guidance on our future product development roadmaps as well as how we can improve our current services and solutions."

Related Link:

• American Libraries article

Digital Preservation Matters.

Oklahoma State University Selects Ex Libris Rosetta for Digital Preservation

Oklahoma State University Chooses Ex Libris Rosetta for Digital Preservation. Press Release. Ex Libris. September 15, 2015.
     Oklahoma State University has adopted the Rosetta digital-asset management and preservation solution. "Rosetta encompasses the entire workflow for managing and preserving digital assets, including their validation, ingest, storage, preservation and delivery. The Rosetta solution handles institutional documents, research output in digital formats, digital images, websites and other digitally-born and digitized materials." The first Rosetta preservation project will be the Oklahoma Oral History Research Program.

The Dean has prioritized the creation of digital content at the OSU Libraries and recognizes the need for long-term preservation and management of these collections. "Rosetta will enable us to develop a sustainable digital preservation program. After evaluating several commercial digital preservation systems, we found Rosetta has the capabilities that we were seeking."

Digital Preservation Matters.

Rosetta Customer Testimonial - Jennifer L. Thoegersen, University of Nebraska–Lincoln

Rosetta Customer Testimonial - Jennifer L. Thoegersen, University of Nebraska–Lincoln. Jennifer Thoegersen. University of Nebraska–Lincoln / Ex Libris. July 5, 2015. [YouTube video.]
      Jennifer Thoegersen, Data Curation Librarian at the University of Nebraska–Lincoln, talks about her experience with using Rosetta for managing and preserving different types of digital content, and its impact at UNL. The challenges that they were facing included having digital materials throughout the library and the campus that they were backing up but they wanted to do more to actively preserve and manage the materials far into the future. Libraries have been tasked to be the gatekeepers for the information. They have lots of different types of content, such as research data, audiovisual content, born digital content, websites, digitized images. They have moved content from ContentDM into Rosetta.

One of the things she really likes about being a Rosetta user is that the Rosetta User Community is very helpful. The group provides insights to working with different types of situations and challenges and they share code as well. The major benefit for UNL is the ability to validate their content, monitor our digital assets over an extended period of time, and being able to tailor the system to meet their needs. Rosetta is an open, extendable, and customizable digital preservation system. The implementation team worked well, and they have also been able too work with the system developers to suggest improvements and have those changes added to the system.

Related posts:

• University of Nebraska–Lincoln Selects Ex Libris Rosetta to Preserve Digital Collections
• Rosetta information
• Rosetta - version 4.2 released

Digital Preservation Matters.

Arizona State University and Northern Arizona University Select Ex Libris Rosetta.

Arizona State University and Northern Arizona University Select Ex Libris Rosetta. Ex Libris Press Release. June 25, 2015.
Arizona State University and Northern Arizona University have adopted the Rosetta digital asset management and preservation solution. Rosetta will enable the libraries to manage and preserve their digital collections, including born-digital objects such as web sites and research data, in perpetuity. With Rosetta, the three institutions will be able to implement the solution together and work off one infrastructure, providing end-to-end digital asset management and preservation for the vast array of assets in all of their libraries.

The two Arizona schools join the University of Arizona, already a Rosetta customer, to provide shared digital asset management and preservation service for all public higher education in the state.

Digital Preservation Matters.