From archival master to usable video with Mark Gray

by Mark Gray
Issue 80 - August 2013 With about 4 million new video channels expected to hit the market in the next five years, its going to be more important than ever to make high-quality content readily available. Many media archives and libraries; studios; broadcasters; production and postproduction facilities; and mastering, quality control, and digital cinema facilities have volumes of pristine master files in their vaults just waiting to be repurposed. The challenge will be making use of their archival masters efficiently. Industry veteran Mark Gray touches on some of the key points.
Some might not grasp the complexity and enormity of archiving; can you share some insight into the process?
Video files represent big data, and in most cases, a lot of time, money, creativity, and expertise go into creating them. These video files can be massive, which requires media organizations to make a significant investment in storing the original, pristine master files, copies of which might go to a postproduction house for manipulation, a studio for previewing, a broadcaster for transcoding, an engineer for test and measurement, an editor for repurposing, or any number of other destinations in the life of a video asset. In any case, archiving the original master files ensures they are preserved and available for later use. In order to ensure high-quality, artifact-free copies, the files are archived in uncompressed or other lossless mastering formats such as JPEG 2000 or DPX. When it comes to viewing and delivering archival master files, what are some of the challenges the industry faces?
Master files are typically stored in archive formats that cannot be played on a standard player or screen, so those who need to work with the files must have expensive, proprietary players for different vendors and file types. In addition, they need professional HD-SDI monitors that can cost tens of thousands of dollars each. Many facilities cannot afford expensive systems with DCP projectors, HD-SDI cards and monitors, and software from each of the manufacturers represented in their media collection for everyone in a group, so many archivists and production teams have to work with lower-quality mezzanine and proxy copies of their masters. This scenario means that those facilities also need storage for those lower-quality files, plus the asset management tools to keep track of them with their masters a system that can cost as much as or more than the storage of the master files themselves.
These challenges are particularly difficult for archives, which often operate without IT support and under incredibly tight budgets that dont allow for expensive, proprietary equipment and software from multiple manufacturers. The result? Many archivists are unable to view the master files in their care. When retrieving a requested file from the archive for delivery or digitizing a tape into a master file for storage, the archivist has no way of knowing whether the copy is faithful to the original or what it will look like on the screen. Why is the process of viewing and delivering master files so complex?
There are several video file formats depending on which vendors hardware (e.g., a camera) or software (e.g., a nonlinear editor) created the video. In turn, those files can be bundled together with codecs, metadata, and other file-based elements to create a package, which is contained inside yet another format, a wrapper, for archiving or transporting. Wrappers including the most common MXF, MOV, and AVI formats hold files from a long list of video essences such as JPEG 2000; DPX (a wrapper/essence format in one); various MPEG-2 and MPEG-4 master profiles; and DCI or DCP for digital cinema, to name just a few. To complicate matters, the video file and wrapper formats arent interchangeable.
On top of the seemingly endless combination of file and wrapper formats, there are nearly as many pieces of third-party equipment on which to play them. Users might need to stream the masters to a host of hardware, including HDMI and SDI monitors, projectors, and multiviewers; editing software; encoders and capture cards; color correctors, restoration workstations, and standards converters; videotape and disk recorders; broadcast playout servers; test equipment such as vectorscopes and waveform monitors; and SDI routers. Theres no guarantee a particular file will play on a particular piece of equipment.
Based on the sheer number of formats and all the possible places to play them, its safe to say that no single format will rise above the rest to become an industry standard anytime soon.
So complexity will continue to present challenges. How do we resolve them?
Master files are the ideal format for converting to other formats, so the industry must adapt solutions that can accommodate those formats. There are no universal players or management tools to support such large and cumbersome files, so most facilities simply store the masters and use other lossy formats when editing or converting files for delivery. Whats needed is an affordable, universal way to work with, interchange, and deliver copies of master files directly from their originals while eliminating the need to store multiple formats. (And by affordable, I mean able to be purchased online using a credit card, with prices and buying experiences like on Amazon.com.) The solution must work for video engineers, archivists, and manufacturers, and allow all of them to see the same things on their standard HDMI HDTV screens without proprietary software or expensive hardware. The ideal solution would support multiple vendors formats and mastering formats in the most popular wrappers from all major manufacturers, thus eliminating the need to work with mezzanine and proxy copies. Instead, users could see their masters with internal technical metadata on their choice of third-party equipment. This revolutionary capability ultimately would make content available more quickly at higher quality, and would make it unnecessary to create and store the content in other formats. As a bonus, it would eliminate distortions by playing files in their native frame rate as opposed to the frame rate in the desktop setting.
Rather than move the people to where the master files can be viewed, or create and store mezzanine and proxy files, or supply everyone with proprietary workstations just for viewing, facilities could simply install an affordable, vendor-neutral product so that users could view master files wherever they work with confidence that theyre seeing the pristine quality thats intended.
Archimedia Technology is new to us. Who are you?
Simply put, we are a new company with ground-breaking technology and applications that bridge the gap between content producers and their archives without loss of quality. My partners and I co-founded Archimedia and created our flagship product, Archimedia Reference Player, to ensure video engineers and archivists can access, play, manage, and test files in mastering formats including JPEG 2000 and DPX. We are explicitly committed to supporting all current and future master formats regardless of origin.
At NAB this year, we saw 4K cameras everywhere, but no way to deliver and read the files. There are no tools to satisfy both the people who create the master files and the people who archive them, so we decided to solve that problem. Weve all spent many years developing video archiving, digitization, and test technologies for some of the worlds leading archive companies, broadcasters, and digital cinema studios. Now weve channeled that expertise into technology that gives users innovative tools theyve never had before but have long needed streamlining an incredibly complex process.