From archival master to usable video with Mark Gray

Mark Gray TV-Bay Magazine
Read ezine online
by Mark Gray
Issue 80 - August 2013
With about 4 million new video channels expected to hit the market in the next five years, its going to be more important than ever to make high-quality content readily available. Many media archives and libraries; studios; broadcasters; production and postproduction facilities; and mastering, quality control, and digital cinema facilities have volumes of pristine master files in their vaults just waiting to be repurposed. The challenge will be making use of their archival masters efficiently. Industry veteran Mark Gray touches on some of the key points.

Some might not grasp the complexity and enormity of archiving; can you share some insight into the process?

Video files represent big data, and in most cases, a lot of time, money, creativity, and expertise go into creating them. These video files can be massive, which requires media organizations to make a significant investment in storing the original, pristine master files, copies of which might go to a postproduction house for manipulation, a studio for previewing, a broadcaster for transcoding, an engineer for test and measurement, an editor for repurposing, or any number of other destinations in the life of a video asset. In any case, archiving the original master files ensures they are preserved and available for later use. In order to ensure high-quality, artifact-free copies, the files are archived in uncompressed or other lossless mastering formats such as JPEG 2000 or DPX.
When it comes to viewing and delivering archival master files, what are some of the challenges the industry faces?

Master files are typically stored in archive formats that cannot be played on a standard player or screen, so those who need to work with the files must have expensive, proprietary players for different vendors and file types. In addition, they need professional HD-SDI monitors that can cost tens of thousands of dollars each. Many facilities cannot afford expensive systems with DCP projectors, HD-SDI cards and monitors, and software from each of the manufacturers represented in their media collection for everyone in a group, so many archivists and production teams have to work with lower-quality mezzanine and proxy copies of their masters. This scenario means that those facilities also need storage for those lower-quality files, plus the asset management tools to keep track of them with their masters a system that can cost as much as or more than the storage of the master files themselves.

These challenges are particularly difficult for archives, which often operate without IT support and under incredibly tight budgets that dont allow for expensive, proprietary equipment and software from multiple manufacturers. The result? Many archivists are unable to view the master files in their care. When retrieving a requested file from the archive for delivery or digitizing a tape into a master file for storage, the archivist has no way of knowing whether the copy is faithful to the original or what it will look like on the screen.
Why is the process of viewing and delivering master files so complex?

There are several video file formats depending on which vendors hardware (e.g., a camera) or software (e.g., a nonlinear editor) created the video. In turn, those files can be bundled together with codecs, metadata, and other file-based elements to create a package, which is contained inside yet another format, a wrapper, for archiving or transporting. Wrappers including the most common MXF, MOV, and AVI formats hold files from a long list of video essences such as JPEG 2000; DPX (a wrapper/essence format in one); various MPEG-2 and MPEG-4 master profiles; and DCI or DCP for digital cinema, to name just a few. To complicate matters, the video file and wrapper formats arent interchangeable.

On top of the seemingly endless combination of file and wrapper formats, there are nearly as many pieces of third-party equipment on which to play them. Users might need to stream the masters to a host of hardware, including HDMI and SDI monitors, projectors, and multiviewers; editing software; encoders and capture cards; color correctors, restoration workstations, and standards converters; videotape and disk recorders; broadcast playout servers; test equipment such as vectorscopes and waveform monitors; and SDI routers. Theres no guarantee a particular file will play on a particular piece of equipment.

Based on the sheer number of formats and all the possible places to play them, its safe to say that no single format will rise above the rest to become an industry standard anytime soon.

So complexity will continue to present challenges. How do we resolve them?

Master files are the ideal format for converting to other formats, so the industry must adapt solutions that can accommodate those formats. There are no universal players or management tools to support such large and cumbersome files, so most facilities simply store the masters and use other lossy formats when editing or converting files for delivery. Whats needed is an affordable, universal way to work with, interchange, and deliver copies of master files directly from their originals while eliminating the need to store multiple formats. (And by affordable, I mean able to be purchased online using a credit card, with prices and buying experiences like on The solution must work for video engineers, archivists, and manufacturers, and allow all of them to see the same things on their standard HDMI HDTV screens without proprietary software or expensive hardware.
The ideal solution would support multiple vendors formats and mastering formats in the most popular wrappers from all major manufacturers, thus eliminating the need to work with mezzanine and proxy copies. Instead, users could see their masters with internal technical metadata on their choice of third-party equipment. This revolutionary capability ultimately would make content available more quickly at higher quality, and would make it unnecessary to create and store the content in other formats. As a bonus, it would eliminate distortions by playing files in their native frame rate as opposed to the frame rate in the desktop setting.

Rather than move the people to where the master files can be viewed, or create and store mezzanine and proxy files, or supply everyone with proprietary workstations just for viewing, facilities could simply install an affordable, vendor-neutral product so that users could view master files wherever they work with confidence that theyre seeing the pristine quality thats intended.

Archimedia Technology is new to us. Who are you?

Simply put, we are a new company with ground-breaking technology and applications that bridge the gap between content producers and their archives without loss of quality. My partners and I co-founded Archimedia and created our flagship product, Archimedia Reference Player, to ensure video engineers and archivists can access, play, manage, and test files in mastering formats including JPEG 2000 and DPX. We are explicitly committed to supporting all current and future master formats regardless of origin.

At NAB this year, we saw 4K cameras everywhere, but no way to deliver and read the files. There are no tools to satisfy both the people who create the master files and the people who archive them, so we decided to solve that problem. Weve all spent many years developing video archiving, digitization, and test technologies for some of the worlds leading archive companies, broadcasters, and digital cinema studios. Now weve channeled that expertise into technology that gives users innovative tools theyve never had before but have long needed streamlining an incredibly complex process.

Tags: iss080 | Archimedia Technology | mark gray | usable video | archiving | process | guide | ask the experts | Mark Gray
Contributing Author Mark Gray

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Facilis at NAB 2014

    Facilis at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Facilis at NAB 2013

    Facilis at NAB 2013

  • TMD at NAB 2013: MediaFlex Reporting Module

    TMD at NAB 2013: MediaFlex Reporting Module

  • TMD at NAB 2013: MediaFlex Systems

    TMD at NAB 2013: MediaFlex Systems

  • TMD at NAB 2013: Content Intelligence

    TMD at NAB 2013: Content Intelligence

  • Object Matrix at BVE 2013

    Object Matrix at BVE 2013

  • Nugen Audio at IBC 2016

    Nugen Audio at IBC 2016

  • Elemental Technologies at IBC 2015

    Elemental Technologies at IBC 2015

  • Cobalt Digital at NAB 2014

    Cobalt Digital at NAB 2014

  • Elemental Technologies Software-Defined Video at NAB 2014

    Elemental Technologies Software-Defined Video at NAB 2014

  • Elemental Technologies HEVC solutions at NAB 2014

    Elemental Technologies HEVC solutions at NAB 2014

  • Elemental Technologies Multiscreen Solutions at NAB 2014

    Elemental Technologies Multiscreen Solutions at NAB 2014

  • Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

    Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

  • Digital Vision on BroadcastShow LIVE at IBC 2013

    Digital Vision on BroadcastShow LIVE at IBC 2013

  • Blue Lucy Media at IBC 2013

    Blue Lucy Media at IBC 2013

  • TC Electronic at IBC2011

    TC Electronic at IBC2011

  • Guided Tour of the Polecam Stand at BVE 2016

    Guided Tour of the Polecam Stand at BVE 2016

  • Guided tour of the Global Distribution Stand at BVE 2016

    Guided tour of the Global Distribution Stand at BVE 2016

  • Guided tour of the Holdan Stand at BVE 2016

    Guided tour of the Holdan Stand at BVE 2016

  • PAG talk about flying with batteries at IBC 2014

    PAG talk about flying with batteries at IBC 2014

  • TMD talk workflow on BroadcastShow LIVE at IBC 2013

    TMD talk workflow on BroadcastShow LIVE at IBC 2013

  • Triveni Digital at NAB 2012

    Triveni Digital at NAB 2012

Related Shows
  • Talking Streaming with Stream UK: BVE Day 3

    Talking Streaming with Stream UK: BVE Day 3

Sony HDC-4800 Review
Andy McKenzie First announced at NAB 2016, the Sony HDC-4800 is a studio camera system capable of shooting 4K/UHD at up to 8x or full HD at up to 16x. With a price point upwards of £250,000 it is a very high-end product with a wide feature set. In Sony's own words, "This is the future of live production, designed to satisfy the storytelling aspect of modern sports production.” Deliveries began in mid 2017 and, after careful preliminary evaluation, we invested in several systems for our hire fleet ahead of the FIFA World Cup in Russia.
Tags: iss134 | review | hdc-4800 | sony | finepoint | Andy McKenzie
Contributing Author Andy McKenzie Click to read or download PDF
Accelerated Workflows with eGPU
Mike Griggs From the UK’s National Trust to magazine publishers to manufacturers, digital content creator Mike Griggs has a wide and varied portfolio of clients for whom he creates 3D art, motion graphics and multimedia exhibits. A typical day might involve sampling birdsong near Virginia Woolf’s country estate or creating 3D animations for VR. To keep on top of these demands, Griggs wanted to take the full power of the GPU computing revolution on the road.
Tags: iss134 | sonnet | egpu | amd | post production | editing | Mike Griggs
Contributing Author Mike Griggs Click to read or download PDF
The brave new world of software based production
Boromy Ung In today’s rapidly evolving broadcast industry, the only constant media organizations can truly count on is change — and the need to adapt as rapidly and cost-effectively as possible. One of the biggest agents of change is the IP revolution, driving broadcasters to migrate their operations to all-software solutions running on commodity, IT-based technologies.
Tags: iss134 | chyronhego | graphics | sports | ott | Boromy Ung
Contributing Author Boromy Ung Click to read or download PDF
What content providers need to know about OTT
Hiren Hindocha As OTT (Over-The-Top) technology has gotten more mature and established robust standards over the years, the concept of OTT monitoring is gaining popularity. With customer expectations soaring, it’s vital for OTT providers to deliver superior quality content. To deliver Quality of Experience (QoE) on par with linear TV broadcast, the entire system, starting from ingest to multi-bitrate encoding to delivery to CDN must be monitored continuously.
Tags: iss134 | ott monitoring | qos | logging | compliance | dash | atsc | cloud | Hiren Hindocha
Contributing Author Hiren Hindocha Click to read or download PDF
State of the Nation - November 2018
Dick Hobbs - new There is an interesting seminar called Size Matters at the KitPlus Show – organised by the publishers of this fine magazine – at MediaCityUK in Salford on 6 November. It’s a talk by cinematographer Alistair Chapman on the way that camera technology is changing, and in particular the size of the electronic device which creates the image is growing.
Tags: iss134 | cmos | 35mm | AJA | Arri | Blackmagic | Canon | Datavideo | GoPro | Grass Valley | Hitachi | Ikegami | JVC | Kinefinity | Nikon | Panasonic | Red | Sony | jpeg2000 | Dick Hobbs - new
Contributing Author Dick Hobbs - new Click to read or download PDF