Checking Your Digital Content: What is Fixity, and When Should I be Checking It?

Checking Your Digital Content: What is Fixity, and When Should I be Checking It? Paula De Stefano, et al. NDSA. October 2014.
     A fundamental goal of digital preservation is to verify that a object has not changed over time or during transfer processes. This is done by checking the “fixity” or stability of the digital content. The National Digital Stewardship Alliance provides this guide to help answer questions about fixity.

Fixity, the property of a digital file or object being fixed or unchanged, is synonymous with bit-level integrity and offers evidence that one set of bits is identical to another. PREMIS defines fixity as "information used to verify whether an object has been altered in an undocumented or unauthorized way." The most widely used tools for fixity are checksums (CRCs) and cryptographic hashes (MD5 and SHA algorithms). Fixity is a tool but by itself it is not sufficient to ensure long-term access to digital information. The fixity information must be used, such as audits of the objects, replacement or repair processes, and other methods to show that the object is or will be understandable. Long term access means the ability to "make sense of and use the contents of the file in the future".

Fixity information helps answer three primary questions:

1. Have you received the files you expected?
2. Is the data corrupted or altered from what you expected?
3. Can you prove the data/files are what you intended and are not corrupt or altered? 

Fixity has other uses and benefits as well, which include:

• Support the repair of corrupt or altered files by knowing which copy is correct 
• Monitor hardware degradation: Fixity checks that fail at high rates may be an indication of media failure.
• Provide confidence to others that the file or object is unchanged
• Meet best practices such as ISO 16363/TRAC and NDSA Levels of Digital Preservation
• Support the monitoring of processes to monitor content integrity as content is moved
• Document provenance and history by maintaining and logging fixity information

Workflows for checking the fixity of digital content includes:

• Generating/Checking Fixity Information on Ingest
• Checking Fixity Information on Transfer
• Checking Fixity at Regular Intervals
• Building Fixity Checking into Storage Systems

Considerations for Fixity Check Frequency include:

• Storage Media: Fixity checks increases media use, which could increase the rate of failure
• Throughput: Your rate of fixity checking will depend on how fast you can run the checks
• Number and Size of Files or Objects: Resource requirements change as the scale of objects increase

Fixity information may be stored in different ways, which will depend on your situation, such as:

• In the object metadata records
• In databases and logs
• Alongside content, such as with BagIt

Digital Preservation Matters.

Do You Have an Institutional Data Policy?

Do You Have an Institutional Data Policy? A Review of the Current Landscape of Library Data Services and Institutional Data Policies. Kristin Briney, Abigail Goben, Lisa Zilinski. Journal of Librarianship and Scholarly Communication. 22 Sep 2015.  [PDF]
     This study was to look at a correlation between policy existence and either library data services or the presence of a data librarian. Data services in libraries are becoming mainstream and librarians have an opportunity to work with researchers at their institutions and help them understand the policies in place or to work toward a policy. Some items of note from the article:

• Fewer universities have a data librarian on staff (37%) than offer data services.
• Many libraries (65%) have a research data repository, either in an IR or in a repository specifically for data. 
• Fewer universities (11%) have dedicated data repositories as compared with IRs that accept data (58%).
• All universities with over $1 billion per year in research expenditures offer data services and a place to host data. Most (89%) of these institutions also have a data librarian. And (33%) have a data repository
• Nearly half (44%) of all universities studied have some type of policy covering research data
• Half of the policies designated an owner of university research data (67%) 
• Data is required to be retained for some period of time (52%)

Standalone data policies covered many topics:

• defined data (61%), 
• identified a data owner (62%), 
• state a specific retention time (62%), 
• identified who can have access to the data (52%), 
• described disposition of the data when a researcher leaves the university (64%) 
• designate a data steward (46%) 

Data services are becoming a standard at major research institutions. However, institutional data policies are often difficult to identify and may be confusing for researchers. The trend of libraries having a data policy, offering data services, and having a data librarian will become typical at major research institutions. 

Digital Preservation Matters.

AWS glitch hits Netflix and Tinder, offers a wake-up call for others. Katherine Noyes. IDG News Service. Sep 21, 2015.
     A number of major websites were affected for a time by glitches in Amazon Web Services' Northern Virginia facility. This is a cautionary lesson to organizations that rely on the cloud service for mission-critical capabilities. The problem resulted in higher-than-normal error rates. Mission-critical systems should have massive redundancies. "In the end, Amazon does not have adequate failover protection, which means its customers need to make sure they do."  Any outage is a significant one for a cloud provider, but "all providers have outages."  More than anything, this is a wake-up call to design your storage to account for problems.

Digital Preservation Matters.

Data Management Practices Across an Institution: Survey and Report

Data Management Practices Across an Institution: Survey and Report. Cunera Buys, Pamela Shaw. Journal of Librarianship and Scholarly Communication. 22 Sep 2015.
     Data management is becoming increasingly important to researchers in all fields. The results of a survey show that both short and long term storage and preservation solutions are needed. When asked, 31% of respondents did not know how much storage they will need, which makes establishing a correctly sized research data storage service difficult. This study presents results from a survey of digital data management practices across all disciplines at a university. In the survey, 65% of faculty said it was important to share data, but less than half of the them "reported that they 'always' or 'frequently' shared their data openly, despite their belief in the importance of sharing".

Researchers produce a wide variety of data types and sizes, but most create no metadata or do not use metadata standards and most researchers were uncertain about how to meet the NSF data management plan requirements (only 45% had a plan). A study in 2011 of data storage and management needs across several academic institutions and found many researchers were satisfied with short-term data storage and management practices, but not satisfied with long-term data storage options. Researchers in the same study did not believe their institutions provided adequate funds, resources, or instruction on good data management practices. When asked about where research data is stored:

• Sixty-six percent use computer hard drives
• 47% use external hard drives
• 50% use departmental or school servers
• 38% store data on the instrument that generated the data
• 31% use cloud-based storage services
•  Dropbox was the most popular service at 63%
• 27% use flash drives
• 6% use external data repositories.

Most researchers expected to store raw and published data, “indefinitely”. Many respondents also selected 5-10 years, and very few said they keep data for less than one year. All schools all schools suggest that data are  relevant for long periods of time or indefinitely. Specific retention preferences by school were:

• The college of arts and sciences prefers “indefinitely” for ALL data types
• Published data: All schools prefer “indefinitely” for published data except
• The law school prefers 1-5 years for published data
• Other data:
• The school of medicine prefers 5-10 years for all other data types
• The school of engineering prefers 1-5 years for all other data types
• The college of arts and sciences “Indefinitely” for raw data
• The school of management “Indefinitely” for raw data

Keeping raw data / source material was useful since researchers may use it for

• future / new studies (77 responses), 
• utilize it for longitudinal studies (9 responses)
• share it with colleagues (6 responses). 
• valuable for replicating study results (10 responses), 
• responding to challenges of published results, 
• data would be difficult or costly to replicate 
• simply stated that it is good scientific practice to retain data (4 responses).

When asked, 66% indicated they would need additional storage; most said 1-500 gigabytes or  “don’t know.” Also, when asked what services would be useful in managing research data the top responses were:

• long term data access and preservation (63%), 
• services for data storage and backup during active projects(60%), 
• information regarding data best practices (58%), 
• information about developing data management plans or other data policies (52%), 
• assistance with data sharing/management requirements of funding agencies (48%), and 
• tools for sharing research (48%).

Since most respondents said they planned to keep their data indefinitely, that means that institutional storage solutions would need to accommodate "many data types and uncertain storage capacity needs over long periods of time". The university studied lacks a long term storage solution for large data, but has short term storage available. Since many researchers store data on personal or laboratory computers, laboratory equipment, and USB drives, there is a greater risk of data loss. There appears to be a need to educate researchers on best practices for data storage and backup.

There appears to be a need to educate researchers on external data repositories that are available and on funding agencies’ requirements for data retention. The library decided to provide a clear set of  funder data retention policies linked from the library’s data management web guide. Long-term storage of data is a problem for researchers because of the data and the lack of stable storage solutions and that limits data retention and sharing.

Digital Preservation Matters.

veraPDF releases prototype validation library for PDF/A-1b

veraPDF releases prototype validation library for PDF/A-1b. News release. veraPDF consortium. 16 September 2015.
     Version 0.4 of the veraPDF validation library is now available. This release delivers a working validation model and validator, an initial, PDF/A-1b validation profile; and a prototype of the PDF feature reporting. This early version allows users to test this implementation of PDF/A-1b validation on single files. The roadmap for 2015 - 2017 is available.

Digital Preservation Matters.

Backblaze releases raw data on all 41,000 HDDs in its data center

Backblaze releases raw data on all 41,000 HDDs in its data center. Lucas Mearian. Computerworld. Feb 4, 2015.
     Backblaze has provided the files of raw data collected over three years on the reliability of over  41,000 hard disk drives in its data center. They have made the information available before, but these are the raw data files that can be used to examine the data in greater detail.  The two files contain the 2013 data and the 2014 data. The 2015 results will be released when they're available.

 "You may download and use this data for free for your own purpose, all we ask is three things 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone, it is free."  Backblaze uses all consumer-class drives to keep its costs down and says they "are as reliable as expensive enterprise-class drives."  The results show "Western Digital's drives lasted an average of 2.5 years, while Hitachi's and Seagate's lasted 2 and 1.4 years, respectively. Even so, some of the individual Hitachi models topped the reliability charts."

Digital Preservation Matters.

Backblaze takes on Google, Amazon with storage at half a penny a gigabyte

Backblaze takes on Google, Amazon with storage at half a penny a gigabyte. Lucas Mearian. Computerworld. September  22, 2015.
     Backblaze announced decreased prices for its new B2 Cloud Storage. This cloud storage service is not encrypted or manipulated in any way, but users can encrypt their own files beforehand. The first 10GB are free then the cost is "half a penny per gigabyte" regardless of the amount of storage. Upload is free and download is $0.05 per GB. The price of $0.005 is less expensive than Amazon's Glacier which also has limits on data retrieval comes with the caveat that retrieving your data may take hours. "We'll offer access via a web GUI. Anyone can access it via our web interface. Then we offer a command line interface for IT people and if you're a developer, there's an API for the service."

Digital Preservation Matters.

Hill Museum & Manuscript Library awarded grant preserving manuscript collections

Hill Museum & Manuscript Library awarded $4 million grant from Arcadia Fund. Press release. St. Johns University. September 22, 2015.
     Arcadia Fund has awarded a grant to the Hill Museum & Manuscript Library (HMML) at Saint John's University for photographic manuscript preservation, which is HMML's core mission. Rev. Columba Stewart said HMML is "currently preserving manuscript collections at sites in Lebanon, Iraq, the Old City of Jerusalem, Egypt, Mali and Malta" and the digital images and related cataloging will be available to scholars throughout the world. HMML has formed partnerships with over 520 libraries and archives to photograph more than 145,000 manuscripts from Europe, Africa, the Middle East and India.

Digital Preservation Matters.

A Selection of Research Data Management Tools Throughout the Data Lifecycle

A Selection of Research Data Management Tools Throughout the Data Lifecycle. Jan Krause. Ecole polytechnique fédérale de Lausanne. September 9, 2015. [PDF]
     This article looks at the data lifecycle management phases and the many tools that exist to help manage data throughout the process. These tools will help researchers make the most out of their data, save time in the long run, promote reproducible research, and minimize the risks with the data. The lifecycle management phases are: discovery, acquisition, analysis, collaboration, writing, publication and deposit in trusted data repositories.  There are tools in each of the areas. A few of the many tools listed are:

• How to Develop a Data Management and Sharing Plan
• DMPonline and DMPTool  to help write data management plans.
• Recommended Data Formats
• ownCloud synchronizes and shares data on several computers 
• re3data global registry of 1,200+ research data repositories in different disciplines

It is important to use appropriated data and metadata standards, especially data formats, which should happen at the beginning since these are difficult to after the project is started.

Digital Preservation Matters.

Taking Control: Identifying Motivations for Migrating Library Digital Asset Management Systems

Taking Control: Identifying Motivations for Migrating Library Digital Asset Management Systems. Ayla Stein, Santi Thompson. D-Lib Magazine. September/October 2015.
     "Digital asset management systems (DAMS) have become important tools for collecting, preserving, and disseminating digitized and born digital content to library patrons." This article looks at why institutions are migrating to other systems and in what direction. Often migrations happen as libraries refine their needs. The literature on the migration process and the implications is limited; this article provide several case studies of repository migration.A presentation by Lisa Gregory "demonstrated the important role digital preservation plays in deciding to migrate from one DAMS to another and reiterated the need for preservation issues and standards to be incorporated into the tools and best practices used by librarians when implementing a DAMS migration".  Repository migration gives institutions the opportunity to move from one type of repository, such as home grown or proprietary, to another type.  Some of the reasons that institutions migrated to other repositories (by those ranked number 1) are:

• Implementation & Day-to-Day Costs
• Preservation
• Extensibility
• Content Management
• Metadata Standards

Formats they wanted in the new system included:

Response	Num.	%
PDF	28	98
JPEG	26	90
MP3	22	76
JPEG2000	21	72
TIFF	21	72
MP4	19	66
MOV	17	59
CSV	16	55
DOC	13	45
DOCX	12	41

For metadata, they wanted the new system to support multiple metadata schema; administrative, preservation, structural, and/or technical metadata standards; local and user created metadata, and linked data. In addition, METS and PREMIS were highly desirable.

The new system should support, among others:

• RDF/XML
• Ability to create modules/plugins/widgets/APIs, etc.  
• Support DOIs and ORCIDs

Preservation features and functionality were the ability to:

• generate checksum values for ingested digital assets.
• perform fixity verification for ingested digital assets.
• assign unique identifiers for each AIP
• support PREMIS or local preservation metadata schema.
• produce AIPs.
• integrate with other digital preservation tools.
• synchronize content with other storage systems (including off site locations).
• support multiple copies of the repository — including dark and light (open and closed) instances.

The survey suggests that "many information professionals are focused on creating a mechanism to ensure the integrity of digital objects." Other curatorial actions were viewed as important, but some "inconclusive results lend further support claims of a disconnect between digital preservation theory and daily practices". About two-thirds were moving to open source repositories, while one fifth were moving to proprietary.

Digital Preservation Matters.

Archiving a digital history

Archiving a digital history: Preserving Penn State’s heritage one link at a time. Katie Jacobs Bohn. Penn State News. September 18, 2015.
     Archivists need a way to preserve digital artifacts so future historians have access to them. This includes content on the internet that can disappear in a short time. Archive-It is a service Penn State archivists are using to make copies of Web pages and arrange them in collections. They want to digitally preserve their cultural heritage, including the University’s academic and administrative information that is published on the Web.

“Web archiving is important because so much of Penn State’s media is ‘born digital,’ or in other words, there’s never a physical copy."  “But we still need a way to keep and preserve this material so it’s not lost forever.” Some quotes from the article:

• Preservation requires more than just backup. 
• "Technology is constantly evolving, and it’s hard to know what digital archiving will look like 50 years from now, let alone hundreds."
• “In the right environment, paper will last hundreds of years, but digital information has a lot of dependencies. To be able to access digital files in the future, you may need a certain kind of hardware and operating system, a compatible version of the software to open the file, not to mention electricity.” 
• “A lot of digital preservation work involves mitigating the risks associated with these dependencies. For example, trying to use open file formats so you don’t need specific software programs that may no longer be around to access them.”

Regardless of what they are trying to preserve, archivists have difficulties with trying to manage the ephemeral nature of culture and history.

Digital Preservation Matters.

Tracing the contours of digital transformation, Part One

Tracing the contours of digital transformation, Part One. September 11, 2015.

     Interesting article about changing our institutions to encompass digital technology.  Some quotes and notes:

• “digital” transformation is, at it’s most fundamental level, not about digital technologies, but about people, mindsets, relationships, and things. 
• transforming our processes will deliver transformed products more effectively
• Delivering innovative (and even revolutionary) experiences is a lot easier to do from a position of knowing what you are (and aren’t) about. 
• there’s still plenty of work to be done to thoughtfully tackle the big issue of digital transformation and become a postdigital institution, "one that has normalized and internalized digital technologies to an extent that they permeate the whole institution and how the institution works".

Digital Preservation Matters.

Enduring Access to Rich Media Content: Understanding Use and Usability Requirements

Enduring Access to Rich Media Content: Understanding Use and Usability Requirements. Madeleine Casad, Oya Y. Rieger and Desiree Alexander. D-Lib Magazine. September 2015.
     Media art has been around for for 40 years and presents serious preservation challenges and obsolescence risks, such as being stored on fragile media. Currently there are no archival best practices for these materials.

• Interactive digital assets are far more complex to preserve and manage than regular files. 
• A single interactive work may contain many media files with different types, formats, applications and operating systems. Any failure and the entire presentation may be unviewable.
• Even a minor problem with can compromise an artwork's "meaning." 
• Migrating information files to another storage medium is not enough to preserve their most important cultural content. 
• Emulation is not always an ideal access strategy since it can introduce additional rendering problems and change the original experience.

The article surveyed media art researchers, curators, and artists, in order to better understand the "relative importance of the artworks' most important characteristics for different kinds of media archives patrons." Some of the problems mentioned were the lack of documentation and metadata, discovery and access, and technical support. Also problems with vanishing webpages, link rot, and poor indexing. 

Artists are concerned about the longevity of their creative work; it can be difficult selling works that may become obsolete within a year.  Curators of new media art may not include born-digital interactive media in their holdings because they are too complex or unsustainable. Some preservation strategies rely on migration, metadata creation, maintaining a media preservation lab, providing climate controlled storage, and collecting documentation from the artists. 

They are also concerned about "authenticity" in a cultural rather than technical sense. InterPARES defines an authentic record as "a record that is what it purports to be and is free from tampering or corruption". With digital art this becomes more difficult to do, since restoring ephemeral, technological or experiential artwork may alter its original form in ways that can affect its meaning. Authenticity may be more of "a sense that the archiving institution has made a good-faith commitment to ensuring that the artist's creative vision has been respected, and providing necessary context of interpretation for understanding that vision—and any unavoidable deviations from it".

Curators need to work with artists to ensure that artworks' most significant properties and interpretive contexts were preserved and documented.  This is more than ensuring bit-level fixity checks or technically accurate renderings of an artwork's contents. The key to digital media preservation is variability, not fixity; finding ways to capture the experience so future generations will get a glimpse of how early digital artworks were created, experienced, and interpreted.

Therefore, diligent curation practices are more essential than ever in order to identify unique or exemplary works, project future use, assess loss risks, and implement cost-efficient strategies.

Digital Preservation Matters.

AWS Storage Update: New Lower Cost S3 Storage Option & Glacier Price Reduction

AWS Storage Update: New Lower Cost S3 Storage Option & Glacier Price Reduction. Jeff Barr. Amazon. 16 September 2015.
      Changes in Amazon pricing and storage options:  Amazon is adding a new storage class for data that is accessed infrequently, the S3 Standard – Infrequent Access, along with Standard and Glacier. This has all of the existing S3 security and access management, data life-cycle policies, cross-region replication, and event notifications features.

Prices for Standard – IA start at $0.0125 / gigabyte / month with a 30 day minimum storage duration and a $0.01 / gigabyte charge for retrieval plus transfer and request charges). Objects smaller than 128 kilobytes are charged for 128 kilobytes of storage. Data life-cycle policies can be defined to move data between Amazon S3 storage classes over time.

Also, the price of Amazon Glacier storage has decreased by up to 36%, based on the region, which is available for as little as $0.007/GB per month.

Digital Preservation Matters.

Challenges facing African music archives

Challenges facing African music archives. Diane Thram. Music In Africa. Sep 15, 2015.
     Music archives in Africa have cultural heritage collections of recordings and musical instruments that  need conservation, dissemination and return to their original communities. Current issues for music heritage archives include sustainability, archiving practice, digital preservation, and internet access. The role of music archives is still emerging. These issues are challenges that few music archives in Africa have been able to meet, largely due to lack of funding and expertise.

The issues of sustainability and the need for funding are perpetual issues faced by most music archives. Creation and publication of audio-visual and print materials, research projects and publications from research, outreach and education, and repatriation projects should all also be part of on-going operations.

Digital Preservation Matters.

Extending OAI-PMH over structured P2P networks for digital preservation

Extending OAI-PMH over structured P2P networks for digital preservation. Everton F. R. Seára, et al. International Journal on Digital Libraries. July 2012.  [PDF]
    An older but interesting article about OAI-PMH and peer to peer networks for preservation. OAI-PMH is a protocol for harvesting metadata from repositories that store digital objects with structured metadata. Since only metadata is shared there is no digital data preservation and digital objects can be easily lost. An alternative to digital preservation of content is to replicate the information in multiple storage repositories for a long-term data preservation environment that ensures reliability and availability of their objects. The article looks at using a proposed OAI-PMH extension to work over a distributed system that can replicate a digital object between nodes in a distributed P2P archiving system.

Digital Preservation Matters.

Important Win for Fair Use

Important Win for Fair Use in ‘Dancing Baby’ Lawsuit. Electronic Frontier Foundation. September 14, 2015.
     A federal court affirmed in a ruling about Lenz v. Universal, often called the “dancing baby” lawsuit, that copyright holders must consider fair use of material sending a copyright takedown notice.  The United States Court of Appeals for the Ninth Circuit ruled that copyright holders must consider fair use before trying to remove content from the Internet. It also rejected the claim that "a victim of takedown abuse cannot vindicate her rights if she cannot show actual monetary loss."

• “Today’s ruling sends a strong message that copyright law does not authorize thoughtless censorship of lawful speech. 
• We’re pleased that the court recognized that ignoring fair use rights makes content holders liable for damages.”

Digital Preservation Matters.

UK Government: What we’re doing on open standards

What we’re doing on open standards. Government Technology Team. 7 September 2015.
     The UK government technology team has been selecting open standards to help government to adapt to changing needs and technologies. "Open Document Format (ODF) is an important standard - by making documents in a format that is open to all, we are ensuring that there are no barriers or bias when we provide services to the public."  The underlying principle is that people should not have to buy new equipment or software in order to read an official document. The technology team looked at the formats that should be used for government documents and chose the Open Document Format. Since then:

• all of central government has committed to moving to the ODF 1.2 for their editable documents
• most departments have published their implementation plans
• the proportion of ODF documents on GOV.UK is increasing steadily
• software suppliers are providing better support for open formats in their products

A guidance manual has been created to help the government departments a they move to ODF. The manual includes topics such as:

• Introduction to Open Document Format (ODF)  
• Procure ODF solutions
• Validators and compliance testing 
• Platforms and devices 
• Accessibility, Privacy and security 
• Best practices and other information

Digital Preservation Matters.

Oklahoma State University Selects Ex Libris Rosetta for Digital Preservation

Oklahoma State University Chooses Ex Libris Rosetta for Digital Preservation. Press Release. Ex Libris. September 15, 2015.
     Oklahoma State University has adopted the Rosetta digital-asset management and preservation solution. "Rosetta encompasses the entire workflow for managing and preserving digital assets, including their validation, ingest, storage, preservation and delivery. The Rosetta solution handles institutional documents, research output in digital formats, digital images, websites and other digitally-born and digitized materials." The first Rosetta preservation project will be the Oklahoma Oral History Research Program.

The Dean has prioritized the creation of digital content at the OSU Libraries and recognizes the need for long-term preservation and management of these collections. "Rosetta will enable us to develop a sustainable digital preservation program. After evaluating several commercial digital preservation systems, we found Rosetta has the capabilities that we were seeking."

Digital Preservation Matters.

EPUB file validator and guidelines

Epubcheck 4.0.0 Available for Download. EPUBZone website. September 8, 2015.
     The latest version of Epubcheck is now available on Github.  This open source tool validates EPUB documents and makes sure they conform to the latest specifications. It is also used to provide validation information at the online idpf validator website and the iBooks Store.

The iBooks site also has support helps for resolving the errors and tips about EPUB namespace and adding alt text to images. Other EPUB guidelines can be found at: EPUB 3 Accessibility Guidelines.

Digital Preservation Matters.