Ask the Experts on Storage and Archive


What is the most important thing to consider when choosing storage for content creation applications?
The single most important aspect to consider when choosing a storage device for content creation is workflow. It is critical to have made a decision with regards to the capture device, the video format to be used and how the finished project will be delivered. This will in turn help you decide on the host interface for the storage system, the required storage capacity, whether you need a RAID protected device or not, if you need to hold content for an extended period of time, and what the best medium is to archive the content on.
How do I calculate the capacity and performance I need from my storage?
There is a wide range of capture devices capable of acquiring data at HD resolutions such as consumer camcorders, DSLR cameras, HD video cameras and 35mm Digital Cinema Cameras. All of these devices use a codec (Compression / Decompression) which is simply an algorithm that shrinks large movie files to a more manageable size, and also makes them playable on a computer. For example, an uncompressed 1080i 10-bit file would have a data rate of 165MB/sec, which is difficult to work with. A codec such as Apple ProRes422 HQ is only 32MB/Sec, REDCODE 42 is 12MB/Sec and AVCHD is 3MB/Sec, reducing both performance and storage requirements. It is also important to consider the number of layers of video being worked with simultaneously. Two layers of ProRes422HQ requires 64MB/Sec of bandwidth, as well as a higher load on the host and hard disks.
Once you have established the capture device and the codec that the data will be acquired in, it is an easy process to extrapolate from the MB/Sec figure to the total storage required. For example, ten hours of ProResHQ 442 would require a storage capacity of approximately 1.12TB. Always keep in mind that a hard disk storage system, no matter what manufacture, will suffer from performance degradation if the storage volume is filled to near capacity. It is also important to make sure that the storage device you choose is capable of sustaining the data rate and I/O required for your chosen codec as failure to do so will lead to a more time consuming editing process or, at worst, the inability to edit the material. There are three key factors that determine the performance of the overall system: the power of the edit workstation, namely the CPU and RAM; the interface to the storage system; and the processing power of the storage system itself, including the number of disks in the system.
What host interface do I need on the storage?
There is a wide range of interfaces on the market from the mass market FireWire 800 (800Mb/sec), USB 2.0 (480Mb/sec), eSATA (3Gb/sec) and the new Thunderbolt (10Gb/sec) connections that provide ‘plug and play’ functionality, to enterprise connectivity such as 8Gb Fibre Channel (8Gb/sec) and 6Gb SAS - Serial Attached SCSI (24Gb/sec) that require a host interface adapter for computers. In general the plug and play interfaces offer a lower performance but are easier to use. The exception to this is the new Thunderbolt interface that provides a staggering 10Gb/sec host interface creating 500-800MB/sec desktop RAIDs that are currently available. As a general rule of thumb the plug and play connectivity types are great for codecs with low data rate requirements and direct attached environments, one storage device, one workstation (or laptop), and a single editor. Connectivity such as Fibre Channel (FC) is generally deployed in larger collaborative workflows with multiple workstations accessing a large amount of centralised storage, which is known as a Storage Area Network (SAN).
A note of caution with regards to interface; just because the interface is capable of making a high performance connection does not mean that the storage device will be capable of providing that amount of performance to the host. The performance of a solution is highly dependent on the number of hard disks in the system. Note to people thinking that a single HDD with a Thunderbolt interface will be the answer to all!
NB do not confuse your megabytes (MB) and megabits (Mb), a megabit is 1/8 of a megabyte and the same goes for gigabytes (GB) and gigabits (Gb)!
What is RAID and do I need it?
RAID is the “Redundant Array of Independent (or Inexpensive) Disks”. If you have been using hard disks for any length of time then I would almost certainly think that you would have had experience of a hard drive failure and as a result might have even lost data. One of the benefits of RAID is that it can be used to protect data.
Common RAID levels are RAID 0, RAID1, RAID 5, and RAID 6, and they all have different properties that give them relative advantages and disadvantages. RAID 0, sometimes referred to as a ‘stripe’, is when data is written across two or more disks. The more disks the data is written over, the more performance gained. However, if one disk fails then the data on the RAID is lost.
RAID 1 is sometimes referred to as a ‘mirror’ and in this process the same data is written to two disks simultaneously, if one of the disks were to fail the data is still intact and readable. RAID 1 provides security but with a performance penalty.
RAID 5 and 6 brings security and performance together, by writing data across three or more disks simultaneously in a similar manner as RAID 0, but it also employs a clever feature called RAID Parity. Parity information is data generated by the RAID engine about the data being written to the RAID set. If a disk fails within the RAID set, the RAID engine can re-create the missing data. RAID 5 incorporates a single layer of parity and can therefore tolerate a single disk failure, while RAID 6 features dual parity information so it can tolerate two disk failures within the RAID set. RAID 5 is now the defacto standard for multi-drive RAID solutions. For example, 8 x 1TB disks in RAID 5 is 7TB and in RAID 6 is 6TB (excluding formatting overhead).
What about shared storage?
Shared storage can be immensely helpful in increasing productivity, allowing multiple users to collaborate on the same projects. Shared storage can also bring benefits in managing media as the media is centrally stored and can be easily managed. There are typically two methods for providing facility-wide shared storage. The first is via a SAN and the second is via Networked Attached Storage (NAS). SAN solutions typically uses FC as a connectivity type as this provides the highest performance access to the centralised storage at a block-level, in other words it tricks the server or workstation (hosts) into thinking it has a single high capacity local disk attached. You are probably familiar with SAN solutions such as Apple XSAN, Tiger Technology metaSAN and Active Storage ActiveSAN. NAS uses ethernet to connect to a centralised storage device at a file level, files copied to and from the NAS are broken up and wrapped up into TCP/IP packets and sent over the network. In general NAS solutions provide lower cost, lower performance storage that is easier to install and manage than a SAN. In addition a SAN requires optical cabling, host adapters, and SAN management software. The advantage of a SAN is high performance, scalability and Quality of Service (QoS) – or the ability to guarantee certain levels of performance to individual hosts. QoS is something more difficult to achieve with a NAS. Other connectivity types for a SAN include iSCSI, FCoE and Infiniband.
I have a tapeless workflow, so how do I keep my media long term?
Creative professionals commonly use FireWire-connected desktop storage devices such as those from Glyph Production Technologies and G-Technology. These are inexpensive and easy to deploy, handle ProResHQ editing and as a result it is becoming common practice for these drives to be used to store media long term as well.
Using hard disks to store data long term is not a good idea, like any mechanical device it will break. When hard disks are powered on they are constantly checking data on the disk and performing error correction. During long periods of inactivity the integrity of the data on the disk is not being checked and corrected if necessary which can lead to degradation of the data on the disk. Additionally inactivity of the disk can often lead to mechanical problems that can cause the disk to fail. RAID systems can be used to keep data on a long term, but there is associated cost in power, cooling and maintenance and the disks at some point will still fail. A note for using SSD for long time storage is that it is still expensive, and likewise if a memory chip fails your data is gone.
So what is the alternative? Tape. To be precise, digital data tape in the form of Linear Tape-Open (LTO). LTO is now in its 5th generation with a native (uncompressed) capacity of 1.5TB per tape and read/write speed of 140MB/sec and a footprint of approximately 10 x 10 x 2 cm. LTO-5 media is very cost effective at under £50 per tape, and other advantages include the robustness and the lifespan of the media (approximately 30 years). The future for LTO is ensured with a road map stretching to LTO-8 at a massive 12.8TB per tape and a read/write speed of 472MB/sec.
Cache-A manufactures cost-effective LTO-5 based archive appliances that make writing to and retrieving from tape easy for photographers, videographers, postproduction and broadcast applications. Cache-A delivers small footprint solutions connected via USB, Firewire and Ethernet with a built-in searchable database to locate which tape the files are on and the location of the tape.
It is possible to scale the idea behind Cache A, by the combination of industry leading tape library manufacturers such as Spectra Logic and software archive products from Atempo and Xendata. The combination of such software and hardware provides highly scalable and efficient archive solutions that require a fraction of the amount of electricity to run, cool and maintain, making the solution a greener and more cost-effective long term archive than disks based systems. It is common to combine high performance disks systems with archive solutions to provide a balanced storage architecture.
In summary, keep your assets safe - if you only have one copy you risk everything. No data, no business.
Global Distribution is a value-added, specialist distributor with a wealth of knowledge and experience in providing storage and infrastructure solutions for data-intensive computing environments within the audio/video, broadcast, CCTV and high-performance computing (HPC) markets, across the UK and EMEA.
www.globaldistribution.com

Tags: storage | archive | redcode | global distribution | esata | raid | iss057 | N/A
Contributing Author N/A

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • Object Based Storage Solutions from Object Matrix at NAB 2017

    Object Based Storage Solutions from Object Matrix at NAB 2017

  • Glyph External Storage including the Studio and Atom raid at IBC 2017

    Glyph External Storage including the Studio and Atom raid at IBC 2017

  • Fibrenetix with StorageDNA at IBC 2014

    Fibrenetix with StorageDNA at IBC 2014

  • Facilis Technology Shared Storage at IBC 2018

    Facilis Technology Shared Storage at IBC 2018

  • All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

    All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

  • Object based Storage Solution from Object Matrix at IBC 2017

    Object based Storage Solution from Object Matrix at IBC 2017

  • Storage DNA at BVE 2016

    Storage DNA at BVE 2016

  • New CEO and news update from TMD at NAB 2017

    New CEO and news update from TMD at NAB 2017

  • SGL Broadcast at IBC 2015

    SGL Broadcast at IBC 2015

  • NOA Audio Solutions at IBC 2013

    NOA Audio Solutions at IBC 2013

  • SGL at NAB 2013

    SGL at NAB 2013

  • Facilis Technology at BVE 2013

    Facilis Technology at BVE 2013

  • Editshare at NAB 2012

    Editshare at NAB 2012

  • Nexsan at NAB 2012

    Nexsan at NAB 2012

  • Arkivum at BVE 2012

    Arkivum at BVE 2012

  • Global Distribution with mLogic at IBC 2013

    Global Distribution with mLogic at IBC 2013

  • Fibrenetix with Quadrus at IBC 2014

    Fibrenetix with Quadrus at IBC 2014

  • Sonnet Technology at IBC2011

    Sonnet Technology at IBC2011

  • Tiger Technology at BVE 2015

    Tiger Technology at BVE 2015

  • Facilis at NAB 2014

    Facilis at NAB 2014

  • GB Labs Space at NAB 2014

    GB Labs Space at NAB 2014

  • ERA at BVE 2014

    ERA at BVE 2014

  • ERA Avere at BVE 2014

    ERA Avere at BVE 2014

  • Front Porch Digital on BroadcastShow LIVE at IBC 2013

    Front Porch Digital on BroadcastShow LIVE at IBC 2013

  • Sonnet Technologies on BroadcastShow LIVE at IBC 2013

    Sonnet Technologies on BroadcastShow LIVE at IBC 2013

  • EditShare on BroadcastShow LIVE at IBC 2013

    EditShare on BroadcastShow LIVE at IBC 2013

  • Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

    Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

  • Aframe Cloud Video at IBC 2013

    Aframe Cloud Video at IBC 2013

  • Facilis at IBC 2013

    Facilis at IBC 2013

  • Facilis at NAB 2013

    Facilis at NAB 2013

  • Object Matrix at BVE 2013

    Object Matrix at BVE 2013

  • Facilis at BVE 2012

    Facilis at BVE 2012

  • Real Life Kit at ProVideo2011

    Real Life Kit at ProVideo2011

  • Suitcase TV at IBC2011

    Suitcase TV at IBC2011

  • ATTO Technology at IBC2011

    ATTO Technology at IBC2011

  • NOA Archive Solutions at IBC 2014

    NOA Archive Solutions at IBC 2014

  • ERA - Cloud Services - at BVE 2015

    ERA - Cloud Services - at BVE 2015

  • SGL at NAB 2014

    SGL at NAB 2014

  • Primestream on BroadcastShow LIVE at IBC 2013

    Primestream on BroadcastShow LIVE at IBC 2013

  • Softron Media Services on BroadcastShow LIVE at IBC 2013

    Softron Media Services on BroadcastShow LIVE at IBC 2013

  • TMD talk asset management solutions on BroadcastShow LIVE at IBC 2013

    TMD talk asset management solutions on BroadcastShow LIVE at IBC 2013

  • Digital Vision on BroadcastShow LIVE at IBC 2013

    Digital Vision on BroadcastShow LIVE at IBC 2013

  • Cinegy at IBC 2013

    Cinegy at IBC 2013

  • Cinegy: Multiviewer at NAB 2013

    Cinegy: Multiviewer at NAB 2013

  • SGL at NAB 2012

    SGL at NAB 2012

  • TMD at BVE 2012

    TMD at BVE 2012

  • FOR-A at IBC2011

    FOR-A at IBC2011

  • Netia at IBC2011

    Netia at IBC2011

  • Guided tour of the Global Distribution Stand at BVE 2016

    Guided tour of the Global Distribution Stand at BVE 2016

  • Global Distribution at BVE 2012

    Global Distribution at BVE 2012

  • Global Distribution at IBC2011

    Global Distribution at IBC2011

  • G-Technologys G Speed Shuttle XL Raid at BVE 2016

    G-Technologys G Speed Shuttle XL Raid at BVE 2016

  • G-Technology at IBC2011

    G-Technology at IBC2011


Related Shows
  • StorageDNA LIVE at BVE 2016

    StorageDNA LIVE at BVE 2016


Articles
Protecting the continuity of transmission
Lorna Garrett Your viewers love you. You consistently bring them their preferred channels 24/7. They’ve come to rely on you for their viewing pleasure. They never miss cheering on their beloved sports teams. They’re the envy of their friends as they watch live concerts of their favourite bands. They gather the family around and catch up on their must-see shows. They don’t have a bad word to say about you.
Tags: iss134 | garland | gpl | streaming | artel | disaster recovery | Lorna Garrett
Contributing Author Lorna Garrett Click to read or download PDF
Accelerated Workflows with eGPU
Mike Griggs From the UK’s National Trust to magazine publishers to manufacturers, digital content creator Mike Griggs has a wide and varied portfolio of clients for whom he creates 3D art, motion graphics and multimedia exhibits. A typical day might involve sampling birdsong near Virginia Woolf’s country estate or creating 3D animations for VR. To keep on top of these demands, Griggs wanted to take the full power of the GPU computing revolution on the road.
Tags: iss134 | sonnet | egpu | amd | post production | editing | Mike Griggs
Contributing Author Mike Griggs Click to read or download PDF
State of the Nation - November 2018
Dick Hobbs - new There is an interesting seminar called Size Matters at the KitPlus Show – organised by the publishers of this fine magazine – at MediaCityUK in Salford on 6 November. It’s a talk by cinematographer Alistair Chapman on the way that camera technology is changing, and in particular the size of the electronic device which creates the image is growing.
Tags: iss134 | cmos | 35mm | AJA | Arri | Blackmagic | Canon | Datavideo | GoPro | Grass Valley | Hitachi | Ikegami | JVC | Kinefinity | Nikon | Panasonic | Red | Sony | jpeg2000 | Dick Hobbs - new
Contributing Author Dick Hobbs - new Click to read or download PDF
Giving Welsh sport a global audience
Adam Amor From the Ospreys Rugby Union team, to the Football Association of Wales, as well as national cycling, swimming and boxing coverage, Port Talbot based Buffoon Film and Media has been heavily involved in putting Welsh sports on the world stage.
Tags: iss134 | blackmagic | atem | buffoon | micro studio camera | Adam Amor
Contributing Author Adam Amor Click to read or download PDF
University and Mental Health
Rhiannon Jenkins University study and mental health has been in the media quite a bit over the last year, and I’m sure there are many people wondering what is going on? The issues are complex, and I suppose the focus of employability off the back of a degree course has raised the stress stakes for a lot of young people. I’m only qualified to talk about this from my perspective, and my story began when I joined a course not knowing I had a mental health condition.
Tags: iss134 | portsmouth uni | mental health | student | tvfutures | Rhiannon Jenkins
Contributing Author Rhiannon Jenkins Click to read or download PDF