The Future of Past Email is PDF

The Future of Past Email is PDF. Chris Prom. Information and Data Manager (IDM). March 6, 2020.
     The article reports on a group of people who look at the question: How should governments, universities, business, and archives ensure the future generations can access and render email? A group looks at ways to capture, preserve, and render. It builds on an earlier report:

The Future of Email Archives: A Report from the Task Force on Technical Approaches for Email Archives. CLIR Publications. August 2018. [PDF

• Email is an increasingly important part of the historical record, yet it is particularly difficult to preserve, putting future access to this vast resource at risk. It looks at what makes email archiving so complex and describes emerging strategies to meet the challenge.

• Addressing the challenges will require commitment from stakeholders, as well as for tool support, testing, and development.

Some institutions preserve emails with MBOX, EML or PST; maintain or emulate old email environments; or transform them to XML. All these ways require a high level of technical support. Others simply store email archives.

The group suggests the PDF format could be used for email, though there are gaps and risks.

• PDF includes data structures that could fully accommodate the diversity of email content and metadata. It is completely self-contained, PDF and designed to capture text and graphical content for archival purposes. 
• Email-to-PDF provides a migration pathway for email messages independent of email applications and could preserve essential attributes of the message.
•     A standardized application of PDF technology could provide source data, universally usable archival-quality renderings including attachments, and provenance metadata.
• It could use existing standards and a diverse vendor community for preserving, searching and reusing email.
• Using PDF could integrate with existing preservation tools for ingesting, storing, preserving and disseminating content from established repository systems already in use in government, academic, public, and corporate archives and libraries.
• Since the PDF format is so widely implemented, there would already be a common understanding of best-practices for archiving email with PDF.

"In short, the "email archiving in PDF" concept seeks to build on widely implemented standards and technologies.  It would allow individuals and institutions a pathway to migrate email into the most widely used format for the distribution of text documents."

Currently there is a drawback for using PDF for email preservation: "attachments, metadata, context, and sometimes, even searchable text are missing. Simply "printing to PDF" fails to meet the specific needs of institutions archiving volumes of complex email messages, at least as currently implemented."  So how can "institutions ensure authenticity, completeness, privacy, security and other needs, especially when working with thousands or millions of messages, when most header metadata and attachments are lost in the conversion?"

The group identified and documented the essential characteristics and technical requirements for converting email into PDF, which will soon be published as a set of fundamental requirements for archiving email.

Digital Preservation Matters.

ePADD 4.0 Released

ePADD 4.0 Final. July 21, 2017.
    This is the latest release of ePADD, a software tool "developed by Stanford University's Special Collections & University Archives that supports archival processes around the appraisal, ingest, processing, discovery, and delivery of email archives."

The software is comprised of four modules:

1. Appraisal: Allows users to gather and review email archives 
2. Processing: Tools to arrange and describe email archives.
3. Discovery: Tools to share a view of email archives with users through web discovery 
4. Delivery: Enables repositories to provide access within a reading room environment.

System Requirements:

• OS: Windows 7 SP1 / 10, Mac OS X 10.10 / 10.11 
• Memory: 8 GB RAM (4 GB RAM allocated to the application by default) 
• Browser: Chrome 50/51, Firefox 47/48 
• Windows installations: Java Runtime Environment 64-bit, 8u101 or later required

ePADD Installation and User Guide
ePADD Github website

Digital Preservation Matters.

Email preservation: How hard can it be?

Email preservation: How hard can it be? Edith Halvarsson. Digital Preservation at Oxford and Cambridge. 7 July, 2017.
      The post summarises highlights of the Digital Preservation Coalition’s briefing on email preservation. What is email? It is "an object, several things and a verb”, a heavily linked and complex object, like the web. "Retention decisions must be made, not only about text content but also about email attachments and external web links. In addition, supporting features (such as instant messaging and calendars) are increasingly integrated into email services and potential candidates for capture."
Email is also a cultural and social practice; capturing relationships and structures of communication is an additional layer to preserve. 

What is being done, or can be done?  Migration is the most common approach to email preservation. EML and Mbox, which is a family of formats, are the most common formats migrated to. They have  different approaches to storing content. Others choose to unpack content which provides a way to display emails and normalise content within them. The emulation approach provides access to content within the original operating environment. Also, ePADD, an open source tool, provides functions for processing and appraisal of Mbox files, but ha other features

There are still questions and issues still to explore, particularly regarding web links. "Email archives may be more valuable to historians as they acquire critical mass".  Some thing that institutions can do are:

• Participate with the  Email Preservation Task Force
• Share your workflows to the Email Preservation Task Force and the community
• Run trial migrations between different email formats such as PST, Mbox and EML and blog about your finding
• Support open source tools such as ePADD and make them sustainable! 

Digital Preservation Matters.

Formation of Task Force for Email Archiving

Mellon Foundation and Digital Preservation Coalition Sponsor Formation of Task Force for Email Archives. Press Release.  November 1, 2016.
      The task force has been created and is charged over the next 12 months to assess current frameworks, tools, and approaches being taken toward these critical historical sources. Personal correspondence is an essential primary source for historians and scholars across and helps "future generations understand and learn from history, providing evidence of the functions and activities of governments, businesses, nonprofit organizations, families, and individuals".  Today's correspondence is digital and emails especially are far more difficult to gather and preserve in an accessible format. "This is a topic of deep concern.  Preserved correspondence helps students of the past develop a nuanced understanding of events, much more so than published or other widely circulated sources." Email has remained resistant preservation efforts at preservation and is currently not systematically acquired by most institutions. 

"As archives include more born-digital collections, the complex technical issues around preserving email are more prevalent and increasingly important. The technical issues around email preservation are compounded by the sheer scale of the collections." Solutions need to be community supported, large-scale with preservation options.  The task force will focus on these three issues:

1. articulating this technical framework, 
2. suggesting how existing tools fit within this framework,
3. beginning to identify any missing elements.

Preservation of email cannot be a single, comprehensive solution, but the "interaction of a variety of solutions covering the entire range of archival activities, from appraising the research value of email to helping researchers discover and use it." It requires an inter-operable toolkit.

Digital Preservation Matters.

A Method for Acquisition and Preservation of Emails

A Method for Acquisition and Preservation of Emails. Claus Jensen, Christen Hedegaard. iPres 2016. (Proceedings p. 72-6/ PDF p. 37-39).
     The paper describes new methods for the acquisition of emails from a broad range of sources not directly connected with the responsible organization, as well ingesting into the repository. Some of the requirements:

Non-technical requirements

• Maximum emulation of traditional paper-based archiving criteria, procedures
• High level of security against loss, degradation, falsification, and unauthorized access
• A library record should exist, even if documents are not publicly available 
• Simple procedure for giving access to third-party by donor
• Maximum degree of auto-archiving
• Minimum degree of curator involvement after Agreement

Technical-oriented requirements

• No new software programs for the donor to learn
• No need for installation of software on the donor’s machine
• As much control over the complete system  as possible
• Automated workflows as much as possible
• Independence from security restrictions on the donor system imposed by others 

New requirements for the second prototype

• The system should be based on standard email components
• Easy to use for both curator and donors
• Donors’ self-deposit
• System based on voluntary/transparent deposit 
• It should be independent of technical platforms  
• Donor ability to transfer emails to the deposit area at any time
• Donor should always have access to donated emails
• Varying levels of access for external use 
• Donors must be able to organize and reorganize emails.
• Donors must be allowed to saved delete emails within a certain time-frame
• The original email header metadata must be preserved
• The donors must be able to deposit other digital content besides emails

The Royal Library created two areas for each donor, the deposit area and the donation area.  The repository supports linked data and uses RDF within its data model that creates relations between the objects. By ingesting the different email representations the system is able to perform file characterization on the email container files, individual emails, and attachments.

"The email project is still active, and there is still time to explore alternative or supplementing methods for the acquisition of emails. Also the task of finding good ways of disseminating the email collections has not yet begun."

Digital Preservation Matters.

Presentations From LC/NARA Symposium on Archiving Email Made Available Online

Presentations From LC/NARA Symposium on Archiving Email Made Available Online. Gary Price. Library Journal. February 2, 2016. [Video and text].
      Videos of six presentations from the “Archiving Email Symposium” held at the Library of Congress June 2015 are now available online. This was a symposium of federal agencies, academic and research libraries, technologists, curators, archivists and records managers who are directly working on collecting and preserving email archives in order to discuss challenges and solutions. The videos are embedded in the article and links point to transcripts.

Archiving Email: Welcome & Introduction  [text]

• Mark Sweeney (Library of Congress)
• Paul Wester (National Archives and Records Administration)

Institutional Approaches to Archiving Email  [text] 

• Ricc Ferrante (Smithsonian Institution Archives)
• Lynda Schmitz Fuhrig (Smithsonian Institution Archives)
• Jaime Schumacher (Northern Illinois University and Digital POWRR Project)

Challenges of Email as a Record  [text]  

• Lisa Haralampus (U.S. Office of the Chief Records Officer)
• Deborah Armentrout (Nuclear Regulatory Commission)
• Jeanette Plante (U.S. Department of Justice)
• Edwin McCeney (U.S. Department of the Interior)

Policy Development for Archiving Email  [text] 

• David Kirsch (University of Maryland)
• Anthony Cocciolo (Pratt Institute School of Information and Library Science)
• Kenneth Hawkins (National Archives and Records Administration)
• Kathleen O’Neill (Library of Congress)
• Margaret McAleer (Library of Congress)
• Christopher Hartten (Library of Congress)

Practical Approaches to Processing Email  [text]  

• Roger Christman (Library of Virginia)
• Aprille Cooke McKay (University of Michigan)
• Dorothy Waugh (Emory University)

Archiving Email: Closing Summary   [text] 

• Chris Prom (University of Illinois Urbana-Champaign)

Digital Preservation Matters.

Cloud-supported preservation of digital papers: A solution for special collections?

Cloud-supported preservation of digital papers: A solution for special collections? Dirk Weisbrod. Liber Quarterly. January 2016.
     A problem for Special Collections is that in many cases digital media have replaced paper for many writers. Digital papers are "difficult to process using established digital preservation strategies, because of their individual and unique nature". The article suggests that document creators should be involved in the preservation process, and that special collections should look at the cloud as a way to solve the problem.

The relatively short durability of digital media is in contrast to the durability of paper. An example in the article shows that data were lost from an Atari computer after a short period of time. Both paper and digital media can be destroyed or damaged, but the potential loss of digital media is much higher since there are many software and hardware components that can fail. The computer skills of the writers can also influence the degree of preservation of the personal digital documents. "To minimize those risks is the task of digital preservation".

A writer’s archive of digital objects (documents, email correspondence, texts, photos, and such) may be scattered over a variety of social networks and web services. This will affect the acquisition of the content by an archive, which would have a problem of identifying and acquiring the digital objects, including accessing the online services, which may be passworded. 

Archives and special collections need to manage these processes for digital preservation and develop a preservation strategy that "matches with the characteristics of digital papers". This needs to change from a “custodial” to a “pre-custodial” view and work with writers and their lifelong personal archives. Writers should contribute to the digital preservation of their own works. Some approaches to consider:

• Regular captures of the creators’ digital data by preservation specialists to be transferred directly into a managed digital repository.
• The periodic transfer of data from old hardware and media to a special collection.
• Have preservation specialists help writers maintain their digital materials
• Provide self archiving of email archives
• IT-supported self-archiving and automated data transfers. The solution could includes services such as
• email archive, like Mailbox
• data storage, like Dropbox 
• website hosting

These approaches could help solve the problem of ongoing archiving while the original objects remain on the creator’s computer and continue to be updated. Another potential problem if writers use cloud services is that accounts may be cancelled if inactive. Archives and Special Collections should consider the cloud not as a problem but as an opportunity to work with authors. "By establishing a cloud, special collections get an instrument that provides writers with a reasonable working environment and, at the same time, enables the preservation of their personal digital archives. The time span between an object’s creation and its preservation, this critical factor of digital preservation, reduces to a minimum."

Digital Preservation Matters.

Yes, We’re Still Talking About Email

Yes, We’re Still Talking About Email. Lynda Schmitz Fuhrig. Smithsonian Institution Archives. August 4, 2015.
     There has been talk that email was going away.Some say that email is being replaced by texting and social media tools, which are options depending on the message content and who it is intended for. But email still has many uses. A business contract is not being sent via Facebook Messenger, and there are still many online forms that require an email address.

Even if email is obsolete in five years (as claimed in an article called "Why Email Will Be Obsolete by 2020") institutions will continue to receive email accounts from previous years that need to be accessible to researchers.

Archives, libraries, museums, universities, and various organizations are exploring email preservation challenges within their collections. Email messages and attachments come from artists, authors, professors, and government officials, to name a few. Researchers, scholars, and journalists have always had an interest in the correspondence from the past. Previously this information was in the printed form of letters, memos, cards, etc.

There are a number of projects that are being developed to provide access and to help preserve email collections. The various options that are being tested and implemented demonstrate that many institutions and organizations understand the importance of preserving email communications from the late 20th and early 21st centuries. Some examples of projects are:

• Virginia's Kaine Email Project
• Stanford’s ePADD or email: Process, Accession, Discovery processes
• University of Maryland email collections from companies that have failed
• Harvard University system email curatorial project

The Smithsonian has been testing an in-house program called DArcMail (Digital Archive Mail System) to provide XML preservation output and a database for searching email messages and attachments within accounts.

Related posts:

• We Welcome Our Email Overlords: Highlights from the Archiving Email Symposium
• The History of Email at the Smithsonian
• Open Source Tools for Records Management
• Digital Tools and Apps
• Preserving Email DPC Technology Watch Report released. 
• Emerging Collaborations for Accessing and Preserving Email
  

 

Digital Preservation Matters.

We Welcome Our Email Overlords: Highlights from the Archiving Email Symposium

We Welcome Our Email Overlords: Highlights from the Archiving Email Symposium. Kate Murray, Erin Engle. The Signal. July 9, 2015.
     Email is becoming an increasingly complex challenge for collecting and memory institutions. There are several different perspectives in looking at archiving emails: technical, archival, records management, and policy and guidelines development. The information about the symposium lists some background reading to learn about email and archiving.

• Keeping and Preserving E-mail. Final Report
• Shaking the Email Format Family Tree 
• InSPECT Significant Properties Testing Report: Electronic Mail
• cc:Mail Archive Email Format
• EML, Email (Electronic Mail Format)

Related posts:

• Why organizations need to archive email   
• Preserving Email DPC Technology Watch Report released.
• Digital Preservation Best Practices and Guidelines.
• Microsoft opens Outlook format, gives programs access to mail, calendar, contacts
• Open Source Tools for Records Management 
• Preserving Email DPC Technology Watch Report released.

Digital Preservation Matters.

Digital Tools and Apps

Digital Tools and Apps. Chris Erickson. Presentation for ULA. 2014. [PDF]
This is a presentation I created for ULA to briefly outline a few tools that I find helpful. There are many useful tools, and more are being created all the time. Here are a few that I use.

• Copy & Transfer Tools: WinSCP; Teracopy;
• Rename Tools: Bulk Rename Utility
• Integrity & Fixity Tools: MD5Summer; MD5sums 1.2; Quick Hash; Hash Tool
• File Editing Tools: Babelpad; Notepad++; XML Notepad; 
• ExifTool; BWF MetaEdit; BWAV Reader;
• File Format Tools: DROID; 
• File Conversion:  Calibre; Adobe Portfolio;
• Others: A whole list of other tools that I use or suggest you look at.
•  PDF/A tools
• Email tools

 Please let me know what tools you find helpful.

Digital Preservation Matters.

ARMA 2014: The Convergence of Records Management and Digital Preservation

ARMA 2014: The Convergence of Records Management and Digital Preservation. Howard Loos, Chris Erickson. October 2014. [PDF]
Presentation on records management and digital preservation given at the ARMA 2014 conference.
Notes:

• Records Management mission: To assist departments in fulfilling their responsibility to identify and manage records and information in accordance with legal, regulatory, and operational requirements
• RIM Life Cycle to DP Life Cycle
• Challenges and successful approaches
• Storing records permanently with M-Discs
• Introduction to Digital Preservation, challenges, format sustainability, media obsolescence, metadata, organizational challenges,
• Life of digital media
• Best practices and processes
• OAIS model
• Rosetta Digital Preservation System
• Library of Congress Digital Preservation Outreach & Education (DPOE) Network

Digital Preservation Matters.

Preserving Email DPC Technology Watch Report released.

Preserving Email DPC Technology Watch Report released. Neil Beagrie.  Blog. News release. 17 Feb 2012.  
The 57 page Preserving Email technology watch report gives practical advice on how to ensure that  email remains accessible.   "Users normally shoulder the ultimate responsibility for managing and preserving their own email.  This exposes important records to needless risks and is counterproductive in many cases." The three basic steps which institutions should undertake:

1. Define policies, including institutional commitment, specific actions to take, and end-user expectations, responsibilities and rights regarding the email archives
2. choosing appropriate tools to work with email in an environment where users have adequate storage space and without auto-deletion settings
3. implementing them in the light of local environmental factors and available resources.

 The report suggests five different preservation strategies with possible tools (such as Aid4Mail and Emailchemy) or methods (co-managing archive folders) in some of the cases, for the institution and individuals.

1. The ‘sweeping up crumbs’ or whole-account approach refers to harvesting email found on a user’s computer or account. 
2. The ‘nurturing and harvesting’ approach, helping email users ensure that critical records are retained in system-neutral formats, then using email migration software to capture and preserve records either as they are created or at the end of a user’s lifetime. An example is providing users a designated ‘archives’ box.
3. The ‘capturing carbon’ or whole system approach implements email archiving software to capture an entire email ecosystem or a portion of that ecosystem to an external email storage environment.
4. The ‘tagging and bagging’ or message-by-message approach, existing electronic record management systems, but which may not be effective.
5. The Personal Archives Service approach, which would offer services such as Carbonite or CrashPlan.

Unless we make the preservation of trusted email records a systematic part of our everyday operations, many important records will be lost. They cite some examples, such as the 22 million emails from the Executive Office of the President of the United States surrounding the Gulf War that we lost when servers were replaced. 

Digital Preservation Matters.

Practical Approaches to Electronic Records: What Works Now (ppt)

Practical Approaches to Electronic Records: What Works Now.  Chris Prom, et al. August 30, 2011.
This is a PowerPoint of a presentation given by several people at SAA.  A few notes from the slides:
Adjectives that are undesirable when describing an archive: Undercounted, undermanaged, inaccessible.
Basic requirements for the digital archive:

• Perform a virus check
• Capture descriptive metadata about the folders and files
• Document the file formats
• Record checksums for the files
• Document the actions taken over time

Atempo Digital Archive Helps Simplify and Streamline Broadcast Workflows With Scalable Storage Integration.

Atempo Digital Archive Helps Simplify and Streamline Broadcast Workflows With Scalable Storage Integration. Press release. Sept. 8, 2011.
Announcement that Atempo Digital Archive (ADA) has been integrated with MediaGrid, to simplify and streamline broadcast workflows. The software addresses long-term data retention requirements and digital preservation. "Atempo enables organizations to preserve and protect digital information simply and effectively, across any infrastructure, on any platform, over long periods of time. Atempo's comprehensive archiving solutions deliver policy-based and workflow-driven management of rich media files, email and other high-value digital assets to maximize the efficiency and performance of storage systems and reduce long-term storage costs."