Showing posts sorted by relevance for query mcgath file. Sort by date Show all posts
Showing posts sorted by relevance for query mcgath file. Sort by date Show all posts

Thursday, May 28, 2015

File identification tools

File identification tools. Gary . File Formats Blog.
Knowing the version and subtype of a file can be important. And knowing the subtypes can indicate whether or not a file is suitable for archiving.

In Linux or Unix the file command can be used to check the file, its identifiers, character encoding, and possibly the language of a text file.  The basic syntax is file --mime filename.

Friday, June 12, 2015

File identification tools, part 3: DROID and PRONOM

File identification tools, part 3: DROID and PRONOM. Gary McGath. File Formats Blog.  June 1, 2015.
DROID (Digital Record Object IDentification) is an open sourced Java-based tool from the UK National Archives that is designed to identify and verify files for digital repositories.  It relies on file format information from the National Archive’s registry, which uses a tool called PRONOM. "DROID depends on files that describe distinctive data values for each format". It can verify single files or large batches of files, or it can be integrated into other applications. DROID generates reports about the file and the identify and verification, or report if it can't match the type of file. Sometimes it may report that a file has more than one matching signature, such as if there is more than one version of a format.


Monday, June 29, 2015

File identification tools, part 5: FITS

File identification tools, part 5: FITS. Gary McGath. File Formats Blog.  June 25, 2015.
The File Information Tool Set (FITS), which aggregates results from several file identification tools, was created by the Harvard University Libraries and is available in Github. FITS uses Apache Tika, DROID, ExifTool, FFIdent, JHOVE, the National Library of New Zealand Metadata Extractor, and four Harvard tools.  The tool can be used in the ingest process; it processes directories and subdirectories, and produces a single XML output file in various schemas. It can be run as a standalone tool or incorporated with other tools, and can be configured to determine which tools to run and which extensions to examine.  Documentation is found on Harvard’s website.

Thursday, June 18, 2015

File identification tools, part 5: JHOVE

File identification tools, part 5: JHOVE. Gary McGath. File Formats Blog. June 11, 2015.
JHOVE is a tool that identifies and validates AIFF, GIF, HTML, JPEG, JPEG2000, PDF, TIFF, WAV, XML, ASCII, and UTF-8 files. Unrecognized files are called a “Bytestream.” 

Key concepts in JHOVE are “well-formed” and “valid.” A file which is “well-formed but not valid” has errors, but not ones that should prevent rendering. JHOVE focuses on the semantics of a file rather than its content.  It only reports full conformance to a profile. It won’t tell you why it fell short.

Download JHOVE from GitHub the Open Preservation Foundation; (do not download from SourceForge). Documentation is on the OPF website. A developer's guide is also available:   JHOVE Tips for Developers.

It shouldn't be confused with JHOVE2 which does similar things but has a different code base.


Monday, July 20, 2015

File identification tools, part 7: Apache Tika

File identification tools, part 7: Apache Tika. Gary McGath. Mad File Format Science Blog.  July 1, 2015.
     Apache Tika is a Java-based open source toolkit that can identify a wide range of formats and extract metadata from others. It doesn’t distinguish variants as much as DROID. Plugins can be added for formats that it does not regularly support.

Related posts:

Thursday, June 18, 2015

File identification tools, part 4: ExifTool

File identification tools, part 4: ExifTool. Gary McGath. File Formats Blog.  June 8, 2015.
The ExifTool, which analyzes Exif metadata, is an open source library and command line tool for identifying, editing, and extracting metadata from many formats, such as image, movie, and audio files. The command options can return metadata in a variety of formats or edit files.

Tuesday, July 21, 2015

File identification tools, part 8: NLNZ Metadata Extraction Tool

File identification tools, part 8: NLNZ Metadata Extraction Tool. Gary McGath. Mad File Format Science Blog.  July 10, 2015.
     This tool is for extracting metadata from files. It uses some basic tests to determine the format and then it looks at the following file formats: 
BMP, GIF, JPEG TIFF, MS Word, Word Perfect, Open Office, MS Works, MS Excel, MS PowerPoint, PDF, WAV, MP3, BWF, FLAC, HTML, XML, and ARC. 
The Java tool is available as open source on SourceForge. There are command line versions for Unix and Windows. [This tool is available to use in Rosetta.]

Related posts:

Friday, June 24, 2016

File-format analysis tools for archivists

File-format analysis tools for archivists. Gary McGath. LWN. May 26, 2016.
     Preserving files for the long term is more difficult than just copying them to a drive. There are other issues are involved. "Will the software of the future be able to read the files of today without losing information? If it can, will people be able to tell what those files contain and where they came from?"

Digital data is more problematic than analog materials, since file formats change. Detailed tools can check the quality of digital documents, analyze the files and report problems. Some concerns:

  • Exact format identification: Knowing the MIME type isn't enough.
  • Format durability: Software can fade into obsolescence if there isn't enough interest to keep it updated.
  • Strict validation: Archiving accepts files in order to give them to an audience that doesn't even exist yet. This means it should be conservative in what it accepts.
  • Metadata extraction: A file with a lot of identifying metadata, such as XMP or Exif, is a better candidate for an archive than one with very little. An archive adds a lot of value if it makes rich, searchable metadata available.
Some open-source applications address these concerns, such as:
  • JHOVE (JSTOR-Harvard Object Validation Environment)
  • DROID and PRONOM
  • ExifTool
  • FITS File Information Tool Set
"Identifying formats and characterizing files is a tricky business. Specifications are sometimes ambiguous."  There are different views on how much error, if any, is acceptable. "Being too fussy can ban perfectly usable files from archives."

"Specialists are passionate about the answers, and there often isn't one clearly correct answer. It's not surprising that different tools with different philosophies compete, and that the best approach can be to combine and compare their outputs"


Saturday, February 06, 2016

MRF for large images

MRF for large images. Gary McGath. Mad File Format Science Blog. January 21, 2016.
NASA, Esri speed delivery of cloud-based imagery data. Patrick Marshall. GCN. Jan 20, 2016.
     NASA and Esri are releasing to the public a jointly developed raster file format and a compression algorithm designed to deliver large volumes of image data from cloud storage.  The format, called MRF (Meta Raster Format) together with a patented compression algorithm called LERC, can deliver online  images ten to fifteen times faster than JPEG2000. The MRF format breaks files into three parts which can be cached separately. The metadata files can be stored those locally so users can "examine data on file contents and download the data-heavy portions only when needed". This would help to minimize the number of files that are transferred. The compression allows users to get faster performance, lower storage requirements, and they estimate the cloud storage costs would be about one-third as much as traditional file-based enterprise storage. An implementation of MRF from NASA is available on GitHub and an implementation of LERC is on GitHub from Esri.

Wednesday, May 13, 2015

Dataliths vs. the digital dark age

Dataliths vs. the digital dark age. Gary McGath. File Formats Blog. May 4, 2015.
Digital technology has allowed us to store more information at less cost than ever before, but in return this information is very fragile in the long term. The chances that your computer’s disk will be readable in a hundred years are poor. Information needs to be stored in a form that can survive long periods of neglect. We need dataliths; this strategy requires "a storage medium which is highly durable and relatively simple to read. It doesn’t have to push the highest edges of storage density. It should be the modern equivalent of the stone tablet, a datalith."

There are devices which tend in this direction. Millenniata, quartz glass data storage, and others. "Hopefully datalith writers will be available before too long, and after a few years they won’t be outrageously expensive. The records they create will be an important part of the long-term preservation of knowledge."


[M-Discs really are the only solution along these lines at present. They are long lived, inexpensive, easy to create and read, and created according to standards. -cle]

Monday, July 06, 2015

TIFF/A

TIFF/A. Gary McGath. File Formats Blog.  July 3, 2015.
   The tiff format has been around for a long time. There have been many changes and additions, such that "TIFF today is the sum of a lot of unwritten rules".  A group of academic archivists have been working on a long term readable version, calling it TIFF/A. A white paper discusses the technical issues. Discussions starting in September will hope to create a version to submit for ISO consideration.

Friday, June 21, 2013

JHOVE 1.10b3

JHOVE 1.10b3. Gary McGath. File Formats Blog.


Thursday, June 18, 2015

Funding for preservation software development

Funding for preservation software development. Gary McGath. File Formats Blog.  June 9, 2015.
The Open Preservation Foundation is launching a new model for funding the development of preservation-related software. The model allows organisations to support individual digital preservation software products and ensure their ongoing sustainability and maintenance. "US libraries have been rather insular in their approach to software development. They’ll use free software if it’s available, but they aren’t inclined to help fund it. If they could each set aside some money for this purpose, it would help assure the continued creation and maintenance of the open source software which is important to their mission."

Monday, November 14, 2016

The (information) machine stops

The (information) machine stops. Gary McGath. Mad File Format Science Blog. March 14, 2016.
     The “Digital Dark Age” discussion comes up again.  Instead of asking what could trigger a Digital Dark Age, we ought to ask
  1. what conditions are necessary and sufficient for the really long-term preservation of information,
  2. what will minimize the risk of widespread loss of today’s history, literature, and news?
Our storage ability has increased but the durability of that storage has decreased. We deal with obsolescence and format, file, and device failures. "Anything we put on a disk today will almost certainly be unusable by 2050. The year 3016 just seems unimaginably far. Yet we still have records today from 1016, 16, and even 984 B.C.E. How can our records of today last a thousand years?"

The current practices rely on curation, migration, and hoping that storage providers will be around forever. Or that some institutions will take up the task of preservation and continue it forever. This requires "an unbroken chain of human activity to keep information alive". History shows that information is often neglected or destroyed, and in reality, only a tiny fraction has survived. "Today’s leading forms of digital storage simply can’t survive that degree of neglect." Abby Smith Rumsey writes, "The new paradigm of memory is more like growing a garden. Everything that we entrust to digital code needs regular tending, refreshing, and periodic migration to make sure that it is still alive, whether we intend to use it in a year, a hundred years, or maybe never." It is not a safe assumption that "things will always be the way they are today, maybe with some gradual improvement or decline, but nothing that will seriously disrupt the way we and future generations live."

However, we have to remember that people and information have survived many types of catastrophes. The original question in the post was "If an uninterrupted succession of custodians isn’t the best way to keep history alive, what is? The answer must be something that’s resilient in the face of interruptions." An important part of this is to avoid reliance on fragile protection; the keys are durability and decentralization. "The hard parts are avoiding physical degradation, hardware obsolescence, and format obsolescence. Physical durability isn’t out of reach. Devices like the M-disc have impressive durability."

"The way to address obsolescence is with designs simple enough that they can be reconstructed." We need decentralized archives in many places with different approaches. "The problem is solvable. The mistake is thinking that an indefinite chain of short-term solutions can add up to a long-term solution."

Related posts:

Wednesday, February 10, 2016

“High-res audio”

“High-res audio”. Gary McGath. Mad File Format Science Blog. February 8, 2016.
    High-res audio, sound digitized at 192,000 samples per second is not necessarily better than the usual 44,000. We can only hear sounds only in a certain frequency range, generally 20 to 20,000 Hertz.

"The sampling rate of a digital recording determines the highest audio frequency it can capture. To be exact, it needs to be twice the highest audio frequency it records." Delivering playback audio at a higher rate offers no benefit and may introduce problems. Another important part of audio is the number of bits per sample, usually 16 bits, but higher-res audio often offers 24 bits. This "isn’t likely to cause any problems", but it doesn't necessarily provide a benefit.   A bigger problem is "over-compressed and otherwise badly processed files".  It is important to not skimp on quality.

Friday, January 15, 2016

DOTS: Almost a datalith

DOTS: Almost a datalith. Gary McGath. Mad File Format Science. December 29, 2015.
     "The notion that archivists will replace outdated digital media every decade or two through the centuries is a pipe dream. Records have always gone through periods of neglect, and they will in the future. Periods of unrest will happen; authorities will try to suppress inconvenient history; groups like Daesh will set out to destroy everything that doesn’t match their worldview; natural disasters will disrupt archiving." DOTS, Digital Optical Technology System, which is burned on tape, can store digital images in any format and also allows them to be recorded as a visual representation. DOTS encodes data physically on an archival tape coated in a phase-change alloy which is resistant to temperature extremes, electromagnetic pulses, and other common environmental hazards. The data, which may include words, images, and digital information, is which is written using a laser that changes the alloy’s index of refraction. "It’s essential to have something like this for reliable long-term data archives. The people who think data will reliably be passed evermore from curator to curator are pleasantly optimistic, but history has never worked that way."

Saturday, December 12, 2015

The FLIF format

The FLIF format. Gary McGath. Mad File Format Science blog. November 25, 2015.
     The post is a look at a new image format FLIF (Free Lossless Image Format) which claims to outcompress other formats for "any kind of image".  "It’s still a work in progress, and any new image format faces an uphill battle"against the existing well-established and well-funded formats. More information about the format is available at the FLIF website. The format is said to be "completely royalty-free and it is not encumbered by software patents." There is still work to do on support for additional metadata and color spaces.

Saturday, August 29, 2015

Update on JHOVE

Update on JHOVE. Gary McGath. Mad File Format Science blog. August 27, 2015.
  Open Preservation Foundation has accepted the stewardship of JHOVE, and Carl Wilson has made impressive progress.  Changes include reorganizing the code and making installation more straightforward.