Datanomics Costs, Benefits, and Value of Research Data

Datanomics: the value of research data. Neil Beagrie. Jisc Invitational Workshop, Glasgo., February 2019.
 Slides from presentation on Datanomics Costs, Benefits, and Value of Research Data. His description of the slides: 

Twenty years ago format obsolescence was seen as the greatest long-term threat to digital information.  Arguably, experience to date has shown that funding and organisational challenges are perhaps more significant threats. I hope this presentation helps those grappling with these challenges and shows some key advances in how to use knowledge of costs, benefits and value to support long-term sustainability of digital data and services.

These are the slides from my keynote presentation to the joint Digital Preservation Coalition / Jisc workshop on Digital Assets and Digital Liabilities - the Value of Data held in Glasgow in February 2018. The slides summarise work over the last decade in the key areas of exploring costs, benefits and value for data. The slides posted here have additional slide notes and references to new publications since the workshop and some modifications such as removal of animations.

Some notes from the slides:

Costs. Keeping Research Data Safe (KRDS)  rules of thumb.

1. Getting data in takes about Half of the lifetime costs, Preservation about a sixth, access about a third. 
2. Preservation costs decline over time. 
3. Fixed costs are significant for most data archives 
4. Staff are the most significant Proportion of archive costs.

The KRDS Benefits Framework. Benefit from Curation of Research Data. Framework arranged on 3 dimensions.
THE ANATOMY OF A BENEFIT Triangle

1. What is the outcome?
2. When is it received?
3. Who benefits?

Valuing Intangible Assets: Measuring value of intangible assets is much harder than for physical assets. We measure value of data services not just data alone

Economic Metrics Used

• Investment Value Amount spent on producing the good or service
• Use Value Amount spent by users to obtain the good or service
• Contingent value: the amount users are “willing to accept” in return for giving up access
• Efficiency gain: user estimates of time saved by using the Data Service resources
• Return on investment: the estimated increase in return on investment due to the additional use

Must also look at the Costs of Inaction

• Rate of loss of research data sets: 17% per annum
• Partial information loss: 7% per annum
• Rate of loss for web-links to data: c. 5.5% per annum
• Access / Data requests fulfilled
• Delay in elapsed time to fulfill data requests. Up to 6 months

Recommendations: Investigate the relative costs and benefits of curation levels, storage, or appraisal for what to keep.

“Five or six decades since the beginning of the Information Age, the namesake of this age, and the major asset driving today’s economy, is still not considered an accounting asset”

“Corporations typically exhibit greater discipline in tracking and accounting for their office furniture than their data”

Conclusions:
Use cost data to look for trends, leverage our efforts, investigate the relative costs and benefits of curation levels, storage, and look towards hierarchical curation management.

Digital Preservation Matters.

Alternatives for Long-Term Storage Of Digital Information

Alternatives for Long-Term Storage Of  Digital Information. Chris Erickson, Barry Lunt. iPres 2015. November 2015.   Poster  Abstract
     This is the poster and abstract that Dr. Lunt and I created and was presented at iPres 2015. The most fundamental component of digital preservation is storing the digital objects in archival repositories. Preservation Repositories must archive digital objects and associated metadata on an affordable and reliable type of digital storage. There are many storage options available; each institution should evaluate the available storage options in order to determine which options are best for their particular needs. This poster examines three criteria in order to help preservationists determine the best storage option for their institution:

1. Cost
2. Longevity
3. Migration Time frame

Each institution may have different storage policies and environments. Not every situation will be the same. By considering the criteria above (the storage costs, the average lifespan of the media and the migration time frame), institutions can make a more informed choice about their archival digital storage environment. The poster has more recent cost information than what is in the abstract.

Digital Preservation Matters.

Digital Curation: think use, not preservation

Digital Curation: think use, not preservation. Jane Stevenson. Archives Hub blog. October 29, 2010.
     From a keynote presentation by Chris Rusbridge about the Blue Ribbon Task Force (BRTF) and  Sustainable Digital Preservation and Access. The most elegant technical solution is no good if it is not sustainable; digital preservation has to be a sustainable economic activity. The focus is on the economic and organizational problems. "It is not just about money; it requires building upon a value proposition, providing incentives to act and defining roles and responsibilities."

No one specifically ‘wants’ preservation; they want access to a resource.  Digital preservation represents a derived demand; it isn't easy to convince someone that they want preservation--you have to sell it on some other basis, such as selling others on the importance of providing use over time. Digital preservation is also ‘path dependent’, which means that your actions and decisions will change over time and they will be different as materials move through the life-cycle. Your actions today can also remove other options for all time.

Often the value of digital preservation is not recognized nor valued, so long-term preservation activities are often funded by short-term allocations. Usually it is not clear who has the responsibility for digital preservation so appropriate organization and oversight is essential for efficient ongoing preservation.

The task force reports that it is essential to:

1. show a compelling value proposition; 
2. provide clear incentives to preserve content; 
3. define preservation roles and responsibilities.

Often people think that digital preservation is expensive, but we must remember that it is in fact relatively cheap when compared to the preservation of physical archives that often require acid-free boxes, rows and rows of shelving, secure, controlled search rooms.  "So, if the cost is actually not prohibitive, and the technical know-how is there, then it seems imperative to address the organisational issues and to really hammer home the true value of preserving our digital data."

Digital Preservation Matters.

Turning a page: downsizing the campus book collections

Turning a page: downsizing the campus book collections. Donald Barclay. The Conversation. August 19, 2015. 
     An article looking at the changing academic libraries and especially the printed book collections. "While I believe there will always be a place for the book in the hearts of academics, it is far less likely there will be a place for the book, or at least for every book, on the academic campus." Keeping a printed book in a library is not cheap. A recent study shows that it costs $4.26 per year to keep a book on the shelf in an open stack collection. The cost of keeping a book in high-density shelving $0.86 per book. This does not mean that the books are disappearing, but that alternative storage solutions are being explored. Books are more in a supporting role rather than the main role.

Digital Preservation Matters.

Keeping Data For A Long Time

Keeping Data For A Long Time. Tom Coughlin. Forbes. June 29, 2014.
     Keeping information for a long time has always been a challenge.  Thermodynamics doesn’t favor information lasting a long time and so to make that happen people have to spend effort and energy. Deciding how to create a long-term archive involves choosing the right storage system with the right technology under the proper environmental conditions.  This can be combined with migration and replication practices to improve the odds of keeping content useful and accessible for an extended period of time. A conference looked at digital storage for long term archiving and preservation.Some of the technologies:

• It appears conventional flash memory may not have good media archive life and should only be used for storing transitory data
• Hard disk drives are used in active archives have problems because they wear out and even if the power is turned off the data in the hard disk drive will eventually decay due to thermal erasure  
• Digital magnetic tape under low temperature/humidity is a good candidate for long-term data retention  
• Optical storage has also been used for long-term data retention and should last at least several decades. Facebook has a 1 PB prototype that should reduce the storage costs by 50% and the energy consumption by 80% of their Hard Disk storage
• Sony said their properly made archival grade optical discs should have a shelf life of 50 years.
• Hitachi Data Systems showed costs for 5 PB of content over 75 years is less than frequent tape and HDD replacement.

A lot of digital data has persistent value and so long term retention of that data is very important. It is estimated the storage for archiving and retention is currently a $3B market growing to over $7B by 2017. "Magnetic tape and optical disks provide low cost long-term inactive storage with additional latency for data access vs. HDDs due to the time to mount the media in a drive.  Thus depending upon the access requirements for an archive it may be most effective to combine two or even three technologies to get the right balance of performance and storage costs."

Related posts:

• How Long Is Long-Term Data Storage? 
• Storage Trends Around Computex 2015
• Preserving Our Digital Heritage

Digital Preservation Matters.

Announcing the ArchivesDirect Price Drop

Announcing the ArchivesDirect Price Drop: Affordable Preservation, Evaluation and Workflows Plus DuraCloud Storage. Carol Minton Morris. DuraSpace. July 21, 2015.
     The ArchivesDirect hosted service from Artefactual Systems and storage in DuraCloud now has reduced pricing. This price includes a hosted instance of Archivematica, training and replicated DuraCloud storage (with a copy in Amazon S3 and one in Amazon Glacier).

The subscription plans are:

1. Assessment. A three month plan with 500 GB of storage. Cost: $4,500
2. Standard. An annual plan with 1 TB of storage. Cost: $9,999
3. Professional. A custom plan. Cost: not available.

Digital Preservation Matters.

Preparing the Workforce for Digital Curation

Preparing the Workforce for Digital Curation. The National Academies Press. 2015.  
This 105 page report focuses on the need for digital curation education and training in order to provide meaningful use of digital information, now and in the future.  [PDF version] This study defines digital curation as: “The active management and enhancement of digital information assets for current and future use.” Digital curation is more than preserving the digital information in secure storage because curation may add value to digital information and increase its utility.

Digital curation is similar to traditional curation. "Regardless of whether a collection is physical or digital, a curator must appraise its value and relevance to the community of potential users; determine the need for preservation; document provenance and authenticity; describe, register, and catalog its content; arrange for long-term storage and preservation; and provide a means for access and use." But it also has many new challenges: the quantities of material to be curated, the need for active and ongoing management, continually changing uses and technology, and the diversity of organizational contexts in which curation occurs. It is more than simply collecting and storing data and information. Active management denotes planned, systematic, coordinated, purposeful, and directed actions that make digital information fit for a purpose. And to ensure that digital information will remain discoverable, accessible, and useable for as long as users have a need and a right to use it.

A new pattern of data usage puts a greater emphasis on the standardization of digital curation practices so that the data can be shared more easily.  Archiving digital data requires a more active management approach, and a more collaborative partnership between producers, archivists and users.

The Loss of Cultural Heritage Through Deterioration of Records and Technological Change: Sound recordings are a striking example of cultural heritage data at high risk of loss. These include music, oral histories, and radio broadcasts preserved in a wide variety of formats and media.

Some benefits of digital curation include:

• Increased collaboration and cost sharing;
• Greater use of data in teaching and research training;
• New opportunities and uses for data, including data mining;
• Creation of a more complete record of research;
• Creation of new areas of research, new industries, or new support services.

Some principal conclusions:

1. Significant opportunities exist to embed digital curation deeply into an organization’s practices to reduce costs and increase benefits. Digital curation will be increasingly in demand across many sectors of society.
2. Digital curation can be advanced by various organizations that can serve as leaders, models, and sources of good curation practices, and build trust by preserving assets.
3. Some barriers to digital curation include: lack of sharing of resources and insufficient resources.
4. There is a need to identify, segregate, and measure the costs of curation tasks in scientific research and business processes.
5. Standards and existing practices vary greatly, which can lead to a lack of coordination across different sectors. This in turn can lead to limited adoption of consistent standards for digital curation and fragmented dissemination of good practices.
6. Automation of at least some digital curation tasks is desirable
7. The knowledge and skills required of those engaged in digital curation are dynamic and highly interdisciplinary.

Some recommendations include:

1. Research communities, educational institutions, and others should work together to develop and adopt digital curation standards and good practices.
2. Work to identify and predict the costs associated with digital curation.
3. Organizations should identify, explain, and measure the benefits derived from digital curation

Digital Preservation Matters.

Getting to the Bottom Line: 20 Cost Questions for Digital Preservation

Getting to the Bottom Line: 20 Cost Questions for Digital Preservation. Matt Schultz. MetaArchive. March 11, 2015.
The MetaArchive Cooperative has, for over a decade, worked to respond transparently to many of the questions they have about the cost of doing digital preservation. These 20 questions will help institutions compare digital preservation solutions. Features and functionality are important, and often easy to identify. But it is more difficult to identify and compare short- and long-term costs, including a variety of  sometimes hidden fees.

The questions are directed at learning about fees, memberships, storage costs, content limits, cost of having the content geographically distributed, hardware and software costs, services and products that the fees pay for, billing plans, and costs for backing up, retrieving or deleting content.

Digital Preservation Matters.

Investing in Curation. A Shared Path to Sustainability. Final RoadMap.

Investing in Curation. A Shared Path to Sustainability. Paul Stokes. The 4C project. March 9, 2015.
Digital curation involves managing, preserving and adding value to digital assets over their entire life cycle. Actively managing digital assets maximizes their value and reduces the risk of obsolescence. The costs of curation is a concern to stakeholders. The final version of the road map is now available; it starts with a focus on the costs of digital curation, but the ultimate goal is to change the way that all organizations manage their digital assets.

The vision: Cost modeling will be a part of the planning and management activities of all digital repositories.

• Identify the value of digital assets and make choices
• Value is an indirect economic determinant on the cost of curating an asset. The perception of value will affect the methods chosen and how much investment is required.
• Content owners should have clear policies regarding the scope of their collections, the type of assets sought, the preferred file formats.
• Establish value criteria for assets as a component of curation, understanding that certain types of assets can be re-generated or re-captured relatively easily, thereby avoiding curation costs
• Demand and choose more efficient systems
• Requirements for curation services should be specified according to accepted standards and best practices.
• More knowledgeable customers demanding better specified and standard functionality means that products can mature more quickly.
• Develop scalable services and infrastructure
• Organizations should aim to work smarter and be able to demonstrate the impact of their investments.
• Design digital curation as a sustainable service
• Effective digital curation requires active management throughout the whole lifecycle of a digital object.
• Curation should be undertaken with a stated purpose.
• Making curation a service further embeds the activity into the organization's normal business function.
• Make funding dependent on costing digital assets across the whole lifecycle
• Digital curation activity requires a flow of sufficient resources for the activity to proceed.
• Some digital assets may need to be preserved in perpetuity but others will have a much more predictable and shorter life-span.
• All stakeholders involved at any point in the curation lifecycle will need to understand their fiscal responsibilities for managing and curating the asset until such time that the asset is transferred to another steward in the lifecycle chain.
• Be collaborative and transparent to drive down costs
• Each organization is looking to realize a return on their investment.
• If those who provide digital curation services can be descriptive about their products and transparent about their pricing structures, this will enhance possible comparisons, drive competitiveness and lead the market to maturity.

Digital Preservation Matters.

Why Digital Storage Formats Are So Risky

Why Digital Storage Formats Are So Risky.  Matthew Woollard.  Lifehacker. 25 February 2015.
While it may seem that digital files last forever, the growing digital sphere faces enormous losses. Even Google has been unable to ensure access for its archive of digital content. Technical solutions already exist, but they’re not well known and relatively expensive.

How much are we prepared to pay to ensure that digital content that exists today will be usable in the future? We need to think about the value of the content and decide if it is worth keeping. Determining the value can be difficult. However, "re-use is a significant benefit from preserving data and adds value." Besides economic value, there are also cultural and intellectual reasons for preserving data. An example of preservation of data from the middle ages can be seen with scribes that used wax tablets for temporary records, and parchment for permanent records.

The chances of born-digital material being usable in 100 years will be considerably improved by actively taking steps now to ensure the preservation of the items. Effective digital preservation relies on the activities of the creator as well as the archivist. It is important to make decisions about providing context, the types of formats to use, how to organize the material, and resolving rights issues to avoid future problems. 

Digital Preservation Matters.

ArchivesDirect hosted service

ArchivesDirect website. February 18, 2015.
ArchivesDirect is a web based hosted service of Archivematica offered by DuraSpace for creating OAIS-based digital preservation workflows with content packages that are archived with DuraCloud and Amazon Glacier. It includes open source preservation tools, and generates archival packets using microservices, PREMIS, and mets xml files. ArchivesDirect is intended for small to mid sized institutions. Duraspace is a partnership with DSpace, Fedora, and Vivo.

Pricing and subscription plans include:
ArchivesDirect Standard (System, training, 1 TB): $11,900
ArchivesDirect Digital Preservation Assessment: $4,500
Additional Storage in Amazon S3 and Glacier: $1,000/TB/year

Digital Preservation Matters.

Preserving progress for future generations

Preserving progress for future generations. Rebecca Pool. Research Information. February/March 2015.
Digital preservation remains one of the most critical challenges facing scholarly communities today. From e-journals and e-books to emails, blogs and more, electronic content is proliferating and organizations worldwide are trying to preserve information before the electronic information is lost. Some of the organizations include: Portico (which preserves content on behalf of participating publishers; the number of open access journals it includes is rising, ); CLOCKSS (still grappling with the cost models of providing preservation service).

There is a rising demand for the preservation of dynamic content. No one is able to "capture dynamic content and [preserve] a day-to-day, or even, minute-to-minute feed of this content." There are only snapshots. CLOCKSS is developing the ‘how to’ process to preserve these ‘snapshots’ across multiple locations, validating each against the other, and is also exploring the best pricing structures to preserve such content.

Other organizations include LOCKSS, The Digital Preservation Network, HathiTrust, Preservica, Archivematica, and Rosetta, whose recent clients are the State Library of New South Wales and the State Library of Queensland.

The digital preservation development is clearly gaining momentum, growing in both size and complexity. "Clearly progress is being made and you can measure that by the maturity of solutions on offer." But for most organizations, the urgency of digital preservation has yet to hit home.

"Trying to sell the idea of digital preservation on the basis of return on investment has been very hard. By its nature, it’s a long-term activity and you’re really hedging your bets against future risks. I think we are still in the very early days of genuinely understanding the value of digital assets... and transferring this understanding over to financial assets doesn’t yet work very well." The European consortium 4C (Collaboration to Clarify the Costs of Curation) has been investigating this problem. Their road map helps organisations appraise digital assets, adopt a strategy to grow preservation assets and develop costing processes. In addition they have developed a model for curation costs. The only way to understand the costs of preservation is though sharing, through openness and collaboration.

Digital Preservation Matters.

Curation Costs Exchange: Supporting Smarter Investments in Digital Curation

Curation Costs Exchange: Supporting Smarter Investments in Digital Curation. Sarah Middleton. Educause Review Online. November 10, 2014.

Tools to manage and estimate costs have not been integrated into other digital curation processes or tools. To determine why that is so a consortium of 13 European cost modeling specialists launched the Collaboration to Clarify the Costs of Curation (4C) project.

4C seeks to help organizations better understand the costs and benefits of digital curation and preservation, and to help users draw together existing and useful resources so they can both make their own assessment of existing models and develop their own cost modeling exercises. The Curation Costs Exchange (CCEx), a platform for the exchange and comparison of digital curation costs and cost information, is a key 4C project deliverable developed to support these goals.

The Cost Comparison Tool enables the exchange of sensitive data and gives users the opportunity to identify greater efficiencies, better practices, and valuable information exchanges among peers. There is also the Understand Your Costs toolkit. The Economic Sustainability Reference model highlights key digital curation concepts, relationships, and decision points in a complex problem space, helping users benchmark and compare their own local models.

Digital Preservation Matters.

DuraCloud Now Offers Low Cost Glacier Storage!

DuraCloud Now Offers Low Cost Glacier Storage!  Carol Minton Morris. Duraspace. May 2, 2013.
DuraCloud has increased the long-term storage options that it offers.  This online storage is intended for durable storage, for data archiving and backup, and particularly for data that is infrequently accessed. It has automatic synchronization between primary and secondary copy, and web access to all copies stored in DuraCloud. Pricing is available at http://duracloud.org/pricing.

Digital Preservation Matters.

The cost of doing nothing

The cost of doing nothing. Barbara Sierman. Digital Preservation Seeds. maart 11, 2013.

There has been much discussion on the fact that over the years the digital preservation community has created more than a dozen cost models, which may increase the confusion in digital preservation even bigger. May be this is part of the way things are going: everyone sees his own situation as something special with special needs.

The solution? Creating an existing model or developing a new one. We can expect help from the recently started European project 4C ,”The Collaboration to Clarify the Costs of Curation”. In their introduction they state that “4C reminds us that the point of this investment [in digital preservation] is to realise a benefit”. Less emphasis on the complexity of digital preservation, and more on the benefits. If we have better figures of the benefits of preserving digital material, we are in a better position to estimate what it will cost us if digital materials are not preserved.

Digital Preservation Matters.