Why is there a growing need for digital archiving in the broadcast and post-production industry?
Across all segments of the media and entertainment industry, the past decade has seen a fundamental transformation, where virtually the entire industry moved from tape or film to digital media. This change required investing in digital equipment and processes: new cameras and new edit workstations, additional servers, storage and software, and hiring or training staff for these new workflows. While the transition to digital is complete for most, the work is far from over. The transition to HD and other higher definition formats is the next wave that has already hit many facilities.
Despite the advantages of digital workflows, there are significant challenges. The most significant is the expanding storage volumes. With each hour of HD footage requiring up to six times more storage capacity than standard definition, the transition from SD to HD is causing content creators to rethink their existing storage architecture. An inadequate storage infrastructure can leave editors scrambling to free up space before they can start editing a new project—or even turn down higher definition projects for lack of storage capacity. Sadly, in many cases the capacity exists, but is tied up in “storage waste,” content that is duplicated within the system as editors swap files and neglect to clean up.
Another challenge is protecting the data. With new high definition digital cameras using more expensive, reusable hard drives or flash as capture media, the natural “backup” that the original filmor videotape provided no longer exists. Without a proper data protection strategy, raw footage is left at risk. A disk crash or accidental deletion can mean a permanent loss of irreplaceable footage.
Also, as digital media workflow technologies emerged, a host of proprietary formats and devices were established with them. Proprietary technologies can hinder media companies from rolling out new technologies and new processes or worse, can create vendor lock-in.
For all these data management challenges, there is a solution—the digital archive.
How is the role of archiving changing with the introduction of digital archives?
Traditionally, archiving was the last step of the creative process. Just before broadcast or distribution, the content would be archived onto a master tape
or film would be stored in a media vault. These vaults were intended as repositories that could be manually searched to retrieve content for later re- use. The harsh reality is that these vaults have been more burial ground than archive.
A well-architected digital archive can be much more than a final resting place. Instead of a final step in the edit process, the archive should be an integral part of the digital media workflow. In a March 2008 seminar, Samuel Reisner of ESPN Sports underscored this concept, stating that: “Any piece of media must be available on any production platform whenever it is needed.”
What are the benefits of a central comprehensive archive in digital media workflows?
From ingest, to edit, to playout or distribution, offering digital archive access to all steps in the workflow can help achieve a range of benefits.
First of all an online digital archive, especially one with rich search and retrieval capabilities, allows for easier content re-use. Previously created content is kept at producers or editors fingertips, making re-use quick and easy. It also empowers content owners to archive and retrieve their projects themselves, minimizing the delays and inefficiency of having archivists acting
as the sole access point to the archive’s content.
Storage growth can also be better managed. Data migration or hierarchical storage management (HSM) features allow administrators to align the storage
performance (and cost) with the access needs and relative value of the content. Making content available doesn’t necessarily mean it needs to be available on high-performance storage. Moving less frequently used content to tape or lower- performance disk can significantly reduce infrastructure costs.
More importantly, the digital archive protects and preserves content throughout its lifespan, from the moment it enters the workflow until decades after the finished work is archived for long-term preservation. The archive not only backs up otherwise unprotected raw footage but also becomes the master vault for historical preservation projects.
With the new expanded role of the archive, what advice would you give to organisations wanting to maximise its potential?
There are several strategies that can help centre the archive in the workflow and maximise its potential.
The first rule is that anyone working with creative content should always archive all raw footage to make sure it will always be available. Many often don’t know what to keep and what to leave on the cutting room floor. However, even the most ordinary clip can become priceless overnight.
Content creators should also be given direct access to the archive. Once the archive is set up, there’s no reason why editors shouldn’t be able to archive and retrieve projects on their own.
Using automatic archiving can significantly reduce clutter in work areas. Despite users’ best intentions, work areas on edit workstations and shared storage areas tend to grow out of control.
Finally, set up multi-tier archives so that more expensive near-line storage can be reserved for content that requires higher performance. Less expensive, off-line “deep” storage can be used for long term preservation or lower priority content. After all, as your archive expands, the reality is that only a subset may be retrieved from the archive.
What should one be looking out for in an archiving solution?
In order to get the best out of your archiving solution, there are several things to look out for that can make a big difference to your workflow and archive management.
An archive solution should be capable of automatically archiving new footage as soon as it enters the workflow, using precise criteria, and attaching metadata to archived files to make it easier and search and locate them in the future.
Choosing a solution with a simple and easy-to-use end-user interface for archive and retrieval, will allow content creators to be able to archive and search themselves.
In order to avoid proprietary workflow components choose a solution that will; provide support for archiving in standard open file format, is integrated with open asset management solutions, offers easy integration with other management tools and supports a variety of archive media and platforms.
To reduce clutter and reclaim waster storage, look for an archive solution that will automatically archive outdated files and once archived, removes files from the work area. Additionally, choose a solution that offers de-duplication to optimise available storage space and does not rely on stubbing as the only retrieval method.
Lastly, pick a solution that supports both near-line and deep archives that can automatically remove files from near-line storage based on criteria such as file age or last access date.
What advice would you give to those undertaking a large-scale preservation?
Those undertaking a large-scale long term preservation project need a robust, reliable archiving solution that will enable long term retention as well as efficient management. The archive solution will need to support high performance configurations to allow massive numbers of files to be archived daily. It will also need to use non-proprietary formats and technologies that support a broad range of media and device types. Cataloguing, indexation and search functionalities should also be a must.