The importance of storage


Larry Jordan# TV-Bay Magazine
Read ezine online
by Larry Jordan
Issue 102 - June 2015

"What's the best computer to edit video?" It's a great question - it's just the wrong question. Yes, CPU speed is important and the Internet is buzzing with the latest specs on GPUs. But, today, virtually any currently shipping computer is more than adequate to edit video; even 4K.

The critical component that no one wants to talk about - because it just isn't very sexy - is storage. Yet, if you don't have the right storage, the fastest computer in the world is not going to work very well. So, let me share some thoughts on what you need to consider when purchasing storage.

SIZE MATTERS - BUT SIZE ISN'T ENOUGH

A few years ago, we debated how many gigabytes to get. Today, we are obsessed with total terabytes. In a year or two, exabytes will be affordable and we'll happily start debating these. The key rule you must keep in mind is that however much storage you buy, it will never be enough.

In Hollywood, there's a saying that an actor "can't be too rich or too thin." With storage, the rule is that a hard disk is "either empty or full." For some reason unknown to the brightest minds on the planet, a half-full hard disk is almost unheard of. Plug-in a brand-new hard disk and, within a couple of days, you are scrambling to delete files to make more space.

Sigh¦

CODECS COUNT

The reason is that video files don't live in a vacuum. The codec that you shoot and edit determines how much space your files will require. For example, the table below illustrates five different codecs all shooting 720p files. The image size, the frame rate is the same, but the file size is not.

Why the difference? Most video formats are compressed and the codec determines the amount of compression. Some formats, like HDV and AVCHD, are massively compressed; removing more than 95% of the original image. Others like RED and ProRes are much less compressed, retaining more of the original image.

So, as you start to budget for storage, you need to first ask yourself which codecs you are likely to be shooting, whether you plan to transcode or render media and what formats you plan to use for final output.

All of these decisions have an impact on the amount of storage you'll need.

YOU NEED CONNECTIONS

Having massive storage is part of the solution, but you also need to give it a fast connection to your computer. In the past, it was a choice between FireWire and USB 2. Neither of those choices are good ones today.

USB 2 is too slow for any video editing; except for transferring files from one drive to another. FireWire is fine for SD, but struggles to keep up with HD. I no longer advise it, unless you are running on an older, slower system without access to faster connections.

Mac users are best advised to use Thunderbolt, with a maximum throughput of 2.2 GB /second. However, as you'll learn in a minute, this number is illusory at best and deceptive at worst. PC users can choose cards that support eSATA, USB3, miniSAS or Thunderbolt. I'm not a PC expert, so I don't have a specific recommendation except to say that USB3 is a safe, middle-of-the-road choice.

THE INTERFACE TRAP

Here's where you need to be careful. Manufacturers are very fond of saying that their equipment "supports Thunderbolt 2 with speeds up to 2.2 GB/second." This is a true statement for all equipment that is approved for Thunderbolt. It is true, but not accurate.
A "spinning-media" hard disk sends data across the wire about 120 MB/second. So, although the "pipe" that connects it runs at 2.2 GB/second, the MOST you can get out of one hard disk is 120 MB/second. Period. Your data is running inside a very large pipe, but it isn't running very fast. This is true of ALL high-speed connections - Thunderbolt, USB3, eSATA, etc. The connection is no longer the gating factor on speed, it's the speed of your hard drives.

"Oh!," I hear you say, "what about solid state drives (SSD)?" SSD drives can be faster than spinning media, but not all SSD drives are equally fast. I've tested SSD drives that were only marginally faster than spinning media. If you are spending lots of money for SSD drives, buy from a reputable company that publishes the actual speeds of their units, not simply states that it supports a particular protocol.

In general, a four SSD array will get close to 1 GB / second. An eight SSD array will get close to 2 GB / second. But there's no free lunch. SSD drives, though potentially MUCH faster than spinning media, hold far less and cost far more. Personally, I use SSD drives for the drive containing the operating system and applications, with an external RAID containing spinning hard drives for all my media. (I like SSDs, but I need storage capacity more than speed.)

RAIDs - TODAY'S SOLUTION

A RAID is a collection of hard drives - either SSD or spinning media - that combine their efforts to create faster speeds and greater storage than a single hard drive alone can support. RAIDs are designed to provide three key benefits:

* Faster speed than a single hard drive
* Greater storage than a single hard drive
* Data "redundancy," meaning that if one of the drives in the RAID dies, all your data is safe.

RAIDs are grouped by RAID "level." There are different levels, but here are the most popular. You buy the RAID level that delivers the performance and features that you need:

RAID 0 - Fast, cheap, no data redundancy. Requires a minimum of two hard drives inside the RAID enclosure. Most often used when speed combined with low cost are paramount. Combines the speed and storage of the drives in the RAID.

RAID 1 - Complete data redundancy. Requires a minimum of two hard drives inside the RAID enclosure. Often called "mirroring," each drive is a complete copy of the other. Most often used for backing up servers or on-set for DIT media work. No faster than the slowest drive in the system.

RAID 5 - Very fast, data redundancy. Requires a minimum of three drives; most often found with four or five drives inside. When one drive dies, your data is still safe. This is the most popular mid-priced RAID for video editing and generally connected to just one computer system.

RAID 6 - Fast, extra data redundancy. Requires a minimum of four drives. This version protects your data in the event two hard drives die at the same time. It used to be more expensive than RAID 5, but, generally, the same physical size. Like the RAID 5 this is most often connected to just one computer. Not as fast as a RAID 5.

RAID 10 - Very fast, totally redundant. Requires a minimum of four drives, but is often created using two matched RAID 0's. This creates a RAID 1 by combining two RAID 0's into one unit. This provides the speed equivalent of a RAID 0, with the data redundancy of RAID 1. As RAIDs continue to drop in price, this can be a less-expensive way to create systems that rival the performance of a RAID 50. Like RAID 5 and RAID 6, this is most often used connected to one computer.

RAIDs vary in price from a few hundred dollars for a RAID 0 to thousands of pound for a 16-drive RAID 10. As with many things in life, greater storage, faster performance or greater redundancy all cost more.

SUMMARY

As you continue your career in media, you'll discover that you will spend FAR more money on storage than you will ever spend on your computer. For this reason, it makes sense to better understand how storage fits into your editing workflow. There are many excellent hardware choices out there - you just need to ask the right questions to get the one that 's best for you.

Tags: iss102 | storage | cpu | 4k | gpu | codec | exabytes | Larry Jordan#
Contributing Author Larry Jordan#

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • Facilis Technology Shared Storage at IBC 2018

    Facilis Technology Shared Storage at IBC 2018

  • All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

    All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

  • Object based Storage Solution from Object Matrix at IBC 2017

    Object based Storage Solution from Object Matrix at IBC 2017

  • Glyph External Storage including the Studio and Atom raid at IBC 2017

    Glyph External Storage including the Studio and Atom raid at IBC 2017

  • Object Based Storage Solutions from Object Matrix at NAB 2017

    Object Based Storage Solutions from Object Matrix at NAB 2017

  • Storage DNA at BVE 2016

    Storage DNA at BVE 2016

  • Fibrenetix with StorageDNA at IBC 2014

    Fibrenetix with StorageDNA at IBC 2014

  • Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

    Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

  • Facilis at IBC 2013

    Facilis at IBC 2013

  • Facilis at NAB 2013

    Facilis at NAB 2013

  • New CEO and news update from TMD at NAB 2017

    New CEO and news update from TMD at NAB 2017

  • SGL Broadcast at IBC 2015

    SGL Broadcast at IBC 2015

  • Tiger Technology at BVE 2015

    Tiger Technology at BVE 2015

  • Fibrenetix with Quadrus at IBC 2014

    Fibrenetix with Quadrus at IBC 2014

  • Facilis at NAB 2014

    Facilis at NAB 2014

  • GB Labs Space at NAB 2014

    GB Labs Space at NAB 2014

  • ERA at BVE 2014

    ERA at BVE 2014

  • ERA Avere at BVE 2014

    ERA Avere at BVE 2014

  • Front Porch Digital on BroadcastShow LIVE at IBC 2013

    Front Porch Digital on BroadcastShow LIVE at IBC 2013

  • Sonnet Technologies on BroadcastShow LIVE at IBC 2013

    Sonnet Technologies on BroadcastShow LIVE at IBC 2013

  • EditShare on BroadcastShow LIVE at IBC 2013

    EditShare on BroadcastShow LIVE at IBC 2013

  • Aframe Cloud Video at IBC 2013

    Aframe Cloud Video at IBC 2013

  • NOA Audio Solutions at IBC 2013

    NOA Audio Solutions at IBC 2013

  • Global Distribution with mLogic at IBC 2013

    Global Distribution with mLogic at IBC 2013

  • SGL at NAB 2013

    SGL at NAB 2013

  • Object Matrix at BVE 2013

    Object Matrix at BVE 2013

  • Facilis Technology at BVE 2013

    Facilis Technology at BVE 2013

  • Editshare at NAB 2012

    Editshare at NAB 2012

  • Nexsan at NAB 2012

    Nexsan at NAB 2012

  • Arkivum at BVE 2012

    Arkivum at BVE 2012

  • Facilis at BVE 2012

    Facilis at BVE 2012

  • Real Life Kit at ProVideo2011

    Real Life Kit at ProVideo2011

  • Suitcase TV at IBC2011

    Suitcase TV at IBC2011

  • Sonnet Technology at IBC2011

    Sonnet Technology at IBC2011

  • ATTO Technology at IBC2011

    ATTO Technology at IBC2011

  • Apantac UHD 4k Multiviewer at NAB 2019

    Apantac UHD 4k Multiviewer at NAB 2019

  • Wireless 4k transmission from BOXX TV at NAB 2019

    Wireless 4k transmission from BOXX TV at NAB 2019

  • Boxx TV Wireless 4K Sub-Frame Breakthrough at NAB 2019

    Boxx TV Wireless 4K Sub-Frame Breakthrough at NAB 2019

  • ADDERLink Infinity 4000 4k KVM at NAB 2019

    ADDERLink Infinity 4000 4k KVM at NAB 2019

  • Compact 4k Cameras CE420-CS, CV506 and CV503 from Marshall at NAB 2019

    Compact 4k Cameras CE420-CS, CV506 and CV503 from Marshall at NAB 2019

  • Lynx Technik Evie 4k HDR to SDR Conversion at NAB 2019

    Lynx Technik Evie 4k HDR to SDR Conversion at NAB 2019

  • Teradek demonstrates the Bolt 4K Wireless Video System at NAB2019

    Teradek demonstrates the Bolt 4K Wireless Video System at NAB2019

  • Matrox Unveils Monarch EDGE 4K Contribution Encoder at IBC 2018

    Matrox Unveils Monarch EDGE 4K Contribution Encoder at IBC 2018

  • 12G 4K microMUSA Jackfield and Fibre Termination Solutions from Canford at IBC 2018

    12G 4K microMUSA Jackfield and Fibre Termination Solutions from Canford at IBC 2018

  • Blackmagic Design Pocket Cinema Camera BMPCC 4K at IBC 2018

    Blackmagic Design Pocket Cinema Camera BMPCC 4K at IBC 2018

  • Apantac 4k UHD HDMI/DVI Multiviewer with KVM at IBC 2018

    Apantac 4k UHD HDMI/DVI Multiviewer with KVM at IBC 2018

  • JVC 4K Connected Camera GY-HC550 and 500 at IBC 2018

    JVC 4K Connected Camera GY-HC550 and 500 at IBC 2018

  • IMT Vislink HCAM 4k UHD transmitter at IBC 2018

    IMT Vislink HCAM 4k UHD transmitter at IBC 2018

  • Blackmagic Pocket Cinema Camera 4K at NAB 2018

    Blackmagic Pocket Cinema Camera 4K at NAB 2018

  • Blackmagic ATEM Television Studio Pro 4K at NAB 2018

    Blackmagic ATEM Television Studio Pro 4K at NAB 2018

  • 4K UHD Multi-viewer HDMI 2.0 from Apantac at NAB 2018

    4K UHD Multi-viewer HDMI 2.0 from Apantac at NAB 2018

  • Omnitek Updates for Ultra 4k Toolbox at NAB 2018

    Omnitek Updates for Ultra 4k Toolbox at NAB 2018

  • 4k and HEVC with Garland at BVE 2018

    4k and HEVC with Garland at BVE 2018

  • 6K Sony Venice and Sony 4k Palm Camcorders at IBC 2017

    6K Sony Venice and Sony 4k Palm Camcorders at IBC 2017

  • JVC 4K UHD Monitors at IBC 2017

    JVC 4K UHD Monitors at IBC 2017

  • The CFU5 4k Transmitter from Vislink at IBC 2017

    The CFU5 4k Transmitter from Vislink at IBC 2017

  • Wireless 4K Transmitter from DTC Domo Broadcast at IBC 2017

    Wireless 4K Transmitter from DTC Domo Broadcast at IBC 2017

  • AJA Thunderbolt 3 IO 4k Plus at IBC 2017

    AJA Thunderbolt 3 IO 4k Plus at IBC 2017

  • HDR, 12G, 4k, IP - the range expands from Phabrix at NAB 2017

    HDR, 12G, 4k, IP - the range expands from Phabrix at NAB 2017

  • Quad 4k Capture and Stream with Maevex from Matrox at NAB 2017

    Quad 4k Capture and Stream with Maevex from Matrox at NAB 2017

  • 4k 60 Matrix from GDSYS at NAB 2017

    4k 60 Matrix from GDSYS at NAB 2017

  • HEVC 4k Encoding from Aviwest at NAB 2017

    HEVC 4k Encoding from Aviwest at NAB 2017

  • Apantac 4k live demo at NAB 2016

    Apantac 4k live demo at NAB 2016

  • Omnitek Ultra 4K Tool Box at IBC 2015

    Omnitek Ultra 4K Tool Box at IBC 2015

  • Snell Advanced Media (SAM) 4k workflow at IBC 2015

    Snell Advanced Media (SAM) 4k workflow at IBC 2015

  • MULTIDYNE Silverback 4k at NAB 2015

    MULTIDYNE Silverback 4k at NAB 2015

  • Atomos talking 4K on BroadcastShow LIVE at IBC 2013

    Atomos talking 4K on BroadcastShow LIVE at IBC 2013

  • Apantac Mi-16 UHD Multiviewer at NAB 2019

    Apantac Mi-16 UHD Multiviewer at NAB 2019

  • Panasonic Lumix GH5 at BVE 2017

    Panasonic Lumix GH5 at BVE 2017

  • Garland at BVE 2017

    Garland at BVE 2017

  • Ikegami 8k camera at IBC 2016

    Ikegami 8k camera at IBC 2016

  • Ikegami HQLM-3120W monitor at IBC 2016

    Ikegami HQLM-3120W monitor at IBC 2016

  • Ikegami UHK-430 camera at IBC 2016

    Ikegami UHK-430 camera at IBC 2016

  • Video Devices PIX-E at IBC 2016

    Video Devices PIX-E at IBC 2016

  • Sound Devices Wingman app at IBC 2016

    Sound Devices Wingman app at IBC 2016

  • JVC GY-LS300 at IBC 2015

    JVC GY-LS300 at IBC 2015

  • AVIWEST at IBC 2015

    AVIWEST at IBC 2015

  • BBright at IBC 2015

    BBright at IBC 2015

  • Thomson Video Networks at IBC 2015

    Thomson Video Networks at IBC 2015

  • Lynx Technik at IBC 2015

    Lynx Technik at IBC 2015

  • JVC at BVE 2015

    JVC at BVE 2015

  • Grass Valley at BVE 2015

    Grass Valley at BVE 2015

  • Gearhouse Broadcast at BVE 2015

    Gearhouse Broadcast at BVE 2015

  • KITPLUS rig setup at IBC 2014

    KITPLUS rig setup at IBC 2014

  • Tektronix at IBC 2014

    Tektronix at IBC 2014

  • Cooke Optics technology at IBC 2014

    Cooke Optics technology at IBC 2014

  • Cooke Optics talk anamorphic at IBC 2014

    Cooke Optics talk anamorphic at IBC 2014

  • JVC GY-LS300 prototype at IBC 2014

    JVC GY-LS300 prototype at IBC 2014

  • MultiDyne SilverBACK at IBC 2014

    MultiDyne SilverBACK at IBC 2014

  • BBright at IBC 2014

    BBright at IBC 2014

  • Hitachi at IBC 2014

    Hitachi at IBC 2014

  • Guntermann and Drunck DP-HR range at IBC 2014

    Guntermann and Drunck DP-HR range at IBC 2014

  • Ateme at IBC 2014

    Ateme at IBC 2014

  • Quantel LiveTouch at IBC 2014

    Quantel LiveTouch at IBC 2014

  • Quantel deal with AFP at IBC 2014

    Quantel deal with AFP at IBC 2014

  • Snell Kahuna Production Switcher at IBC 2014

    Snell Kahuna Production Switcher at IBC 2014

  • Matrox Monarch HD at NAB 2014

    Matrox Monarch HD at NAB 2014

  • Elemental Technologies Software-Defined Video at NAB 2014

    Elemental Technologies Software-Defined Video at NAB 2014

  • Elemental Technologies HEVC solutions at NAB 2014

    Elemental Technologies HEVC solutions at NAB 2014

  • Elemental Technologies Multiscreen Solutions at NAB 2014

    Elemental Technologies Multiscreen Solutions at NAB 2014

  • Guntermann and Drunck at NAB 2014

    Guntermann and Drunck at NAB 2014

  • Matrox Mojito at NAB 2014

    Matrox Mojito at NAB 2014

  • Matrox Capture Card at NAB 2014

    Matrox Capture Card at NAB 2014

  • Blackmagic URSA at NAB 2014

    Blackmagic URSA at NAB 2014

  • Blackmagic Studio Camera at NAB 2014

    Blackmagic Studio Camera at NAB 2014

  • Cintel Film Scanner: Blackmagic Design at NAB 2014

    Cintel Film Scanner: Blackmagic Design at NAB 2014

  • Guntermann and Drunck at BVE 2014

    Guntermann and Drunck at BVE 2014

  • Philip Bloom on Tour in Belfast

    Philip Bloom on Tour in Belfast

  • Cinegy on BroadcastShow LIVE at IBC 2013

    Cinegy on BroadcastShow LIVE at IBC 2013

  • Blackmagic Design on BroadcastShow LIVE at IBC 2013

    Blackmagic Design on BroadcastShow LIVE at IBC 2013

  • Miranda on BroadcastShow LIVE at IBC 2013

    Miranda on BroadcastShow LIVE at IBC 2013

  • Snell on BroadcastShow LIVE at IBC 2013

    Snell on BroadcastShow LIVE at IBC 2013

  • JVC GY-HM600 series camera updates at IBC 2013

    JVC GY-HM600 series camera updates at IBC 2013

  • ATEME at IBC 2013

    ATEME at IBC 2013

  • Multidyne SilverBack, BullDog and Spider at IBC 2013

    Multidyne SilverBack, BullDog and Spider at IBC 2013

  • Cinegy at IBC 2013

    Cinegy at IBC 2013

  • Blackmagic Design with the new ATEM Production Suite at IBC 2013

    Blackmagic Design with the new ATEM Production Suite at IBC 2013

  • MultiDyne: SilverBACK at NAB 2013

    MultiDyne: SilverBACK at NAB 2013

  • Quantel Pablo Rio at NAB 2013

    Quantel Pablo Rio at NAB 2013

  • Blackmagic Design: Products at NAB 2013

    Blackmagic Design: Products at NAB 2013

  • AJA at BVE 2013

    AJA at BVE 2013

  • JVC at BVE 2012

    JVC at BVE 2012

  • GPU acceleration from Sonnet Technologies with Adobe support at NAB 2019

    GPU acceleration from Sonnet Technologies with Adobe support at NAB 2019

  • External GPU Enclosures from Sonnet at IBC 2017

    External GPU Enclosures from Sonnet at IBC 2017

  • Mac Mini support from Sonnet Technologies at NAB 2019

    Mac Mini support from Sonnet Technologies at NAB 2019

  • Elemental Technologies at IBC2011

    Elemental Technologies at IBC2011

  • Cinegy Daniel 2 Codec shown at IBC 2018

    Cinegy Daniel 2 Codec shown at IBC 2018

  • Glensound Dante at IBC 2014

    Glensound Dante at IBC 2014

  • JVC GY-HM650 upgrade at NAB 2013

    JVC GY-HM650 upgrade at NAB 2013

  • ATOMOS at BVE North 2012

    ATOMOS at BVE North 2012

  • Telestream at NAB 2012

    Telestream at NAB 2012

  • Prodys at IBC2011

    Prodys at IBC2011


Related Shows
  • StorageDNA LIVE at BVE 2016

    StorageDNA LIVE at BVE 2016


Articles
IMF Adoption
Howard Twine The Interoperable Master Format isn’t new. It’s been around for more than six years. Despite this longevity, the broad adoption since the publication of SMPTE 2067 has been pretty slow. This is perhaps more because of the age old mantra “if it ain’t broke don’t fix it,” than the ease with which many could interpret the standard. However, it is safe to say that movie producers blazed a trail in using IMF for original content delivery. Perhaps the key is that those early adopters didn’t have any significant tape based infrastructure that had been modified for file-based working when receiving content. So, many longer established content distributors have the immense resources to create the vast array of versions required to address the ‘version issue.’ This being how do you get the same content to look good on the back of an aeroplane seat headrest or an iPhone in the back of car as well as a good old TV set. Over time these resources have been diminishing and many organisations recognise that they have to adopt workflow automation to survive.
Tags: iss137 | editshare | imf | standards | smpte 2067 | Howard Twine
Contributing Author Howard Twine Click to read or download PDF
Preparing for the holiday peak
Jamie Adkin The summer holiday getaway of 2018 hit the headlines for all the wrong reasons. A UK airport's non-functional departure boards left travellers unable to find their flights. Meanwhile, an airline cancelled over 2,000 flights due to a computer glitch. One European city closed its airspace entirely.
Tags: iss137 | adder | kvm | server | Jamie Adkin
Contributing Author Jamie Adkin Click to read or download PDF
Canon EOS C300 MkII Camera Review
Andy McKenzie Canon described the EOS C300 Mark II at its launch as 'a game-changer for creative film-making'. What interested me was whether the camera's impressive feature set, compact size and very aggressive price point matched the needs of the video production market.
Tags: iss137 | finepoint | c300 review | canon c300 | Andy McKenzie
Contributing Author Andy McKenzie Click to read or download PDF
Capturing Haute Couture
KitPlus As the fashion world returned to the French capital for the most prestigious of the 'Big Four’ fashion weeks, historical venues such as the Carrousel du Louvre and the Grand Palais played host to famous emblematic brands such as Chanel, Valentino, Louis Vuitton and Stella McCartney, showcasing their autumn/winter 2019 collections.
Tags: iss137 | blackmagic design | ursa mini | freecaster | atem | KitPlus
Contributing Author KitPlus Click to read or download PDF
Behind the Scenes of Wild rose
KitPlus Financed by BFI Film Fund, Wild Rose tells the story of a musician from Glasgow who dreams of becoming a Nashville star. The only thing in the way: her own penchant for crime and violence.
Tags: iss137 | blackmagic design | davinci | ursa mini pro | KitPlus
Contributing Author KitPlus Click to read or download PDF