The importance of storage


Larry Jordan# TV-Bay Magazine
Read ezine online
by Larry Jordan
Issue 102 - June 2015

"What's the best computer to edit video?" It's a great question - it's just the wrong question. Yes, CPU speed is important and the Internet is buzzing with the latest specs on GPUs. But, today, virtually any currently shipping computer is more than adequate to edit video; even 4K.

The critical component that no one wants to talk about - because it just isn't very sexy - is storage. Yet, if you don't have the right storage, the fastest computer in the world is not going to work very well. So, let me share some thoughts on what you need to consider when purchasing storage.

SIZE MATTERS - BUT SIZE ISN'T ENOUGH

A few years ago, we debated how many gigabytes to get. Today, we are obsessed with total terabytes. In a year or two, exabytes will be affordable and we'll happily start debating these. The key rule you must keep in mind is that however much storage you buy, it will never be enough.

In Hollywood, there's a saying that an actor "can't be too rich or too thin." With storage, the rule is that a hard disk is "either empty or full." For some reason unknown to the brightest minds on the planet, a half-full hard disk is almost unheard of. Plug-in a brand-new hard disk and, within a couple of days, you are scrambling to delete files to make more space.

Sigh¦

CODECS COUNT

The reason is that video files don't live in a vacuum. The codec that you shoot and edit determines how much space your files will require. For example, the table below illustrates five different codecs all shooting 720p files. The image size, the frame rate is the same, but the file size is not.

Why the difference? Most video formats are compressed and the codec determines the amount of compression. Some formats, like HDV and AVCHD, are massively compressed; removing more than 95% of the original image. Others like RED and ProRes are much less compressed, retaining more of the original image.

So, as you start to budget for storage, you need to first ask yourself which codecs you are likely to be shooting, whether you plan to transcode or render media and what formats you plan to use for final output.

All of these decisions have an impact on the amount of storage you'll need.

YOU NEED CONNECTIONS

Having massive storage is part of the solution, but you also need to give it a fast connection to your computer. In the past, it was a choice between FireWire and USB 2. Neither of those choices are good ones today.

USB 2 is too slow for any video editing; except for transferring files from one drive to another. FireWire is fine for SD, but struggles to keep up with HD. I no longer advise it, unless you are running on an older, slower system without access to faster connections.

Mac users are best advised to use Thunderbolt, with a maximum throughput of 2.2 GB /second. However, as you'll learn in a minute, this number is illusory at best and deceptive at worst. PC users can choose cards that support eSATA, USB3, miniSAS or Thunderbolt. I'm not a PC expert, so I don't have a specific recommendation except to say that USB3 is a safe, middle-of-the-road choice.

THE INTERFACE TRAP

Here's where you need to be careful. Manufacturers are very fond of saying that their equipment "supports Thunderbolt 2 with speeds up to 2.2 GB/second." This is a true statement for all equipment that is approved for Thunderbolt. It is true, but not accurate.
A "spinning-media" hard disk sends data across the wire about 120 MB/second. So, although the "pipe" that connects it runs at 2.2 GB/second, the MOST you can get out of one hard disk is 120 MB/second. Period. Your data is running inside a very large pipe, but it isn't running very fast. This is true of ALL high-speed connections - Thunderbolt, USB3, eSATA, etc. The connection is no longer the gating factor on speed, it's the speed of your hard drives.

"Oh!," I hear you say, "what about solid state drives (SSD)?" SSD drives can be faster than spinning media, but not all SSD drives are equally fast. I've tested SSD drives that were only marginally faster than spinning media. If you are spending lots of money for SSD drives, buy from a reputable company that publishes the actual speeds of their units, not simply states that it supports a particular protocol.

In general, a four SSD array will get close to 1 GB / second. An eight SSD array will get close to 2 GB / second. But there's no free lunch. SSD drives, though potentially MUCH faster than spinning media, hold far less and cost far more. Personally, I use SSD drives for the drive containing the operating system and applications, with an external RAID containing spinning hard drives for all my media. (I like SSDs, but I need storage capacity more than speed.)

RAIDs - TODAY'S SOLUTION

A RAID is a collection of hard drives - either SSD or spinning media - that combine their efforts to create faster speeds and greater storage than a single hard drive alone can support. RAIDs are designed to provide three key benefits:

* Faster speed than a single hard drive
* Greater storage than a single hard drive
* Data "redundancy," meaning that if one of the drives in the RAID dies, all your data is safe.

RAIDs are grouped by RAID "level." There are different levels, but here are the most popular. You buy the RAID level that delivers the performance and features that you need:

RAID 0 - Fast, cheap, no data redundancy. Requires a minimum of two hard drives inside the RAID enclosure. Most often used when speed combined with low cost are paramount. Combines the speed and storage of the drives in the RAID.

RAID 1 - Complete data redundancy. Requires a minimum of two hard drives inside the RAID enclosure. Often called "mirroring," each drive is a complete copy of the other. Most often used for backing up servers or on-set for DIT media work. No faster than the slowest drive in the system.

RAID 5 - Very fast, data redundancy. Requires a minimum of three drives; most often found with four or five drives inside. When one drive dies, your data is still safe. This is the most popular mid-priced RAID for video editing and generally connected to just one computer system.

RAID 6 - Fast, extra data redundancy. Requires a minimum of four drives. This version protects your data in the event two hard drives die at the same time. It used to be more expensive than RAID 5, but, generally, the same physical size. Like the RAID 5 this is most often connected to just one computer. Not as fast as a RAID 5.

RAID 10 - Very fast, totally redundant. Requires a minimum of four drives, but is often created using two matched RAID 0's. This creates a RAID 1 by combining two RAID 0's into one unit. This provides the speed equivalent of a RAID 0, with the data redundancy of RAID 1. As RAIDs continue to drop in price, this can be a less-expensive way to create systems that rival the performance of a RAID 50. Like RAID 5 and RAID 6, this is most often used connected to one computer.

RAIDs vary in price from a few hundred dollars for a RAID 0 to thousands of pound for a 16-drive RAID 10. As with many things in life, greater storage, faster performance or greater redundancy all cost more.

SUMMARY

As you continue your career in media, you'll discover that you will spend FAR more money on storage than you will ever spend on your computer. For this reason, it makes sense to better understand how storage fits into your editing workflow. There are many excellent hardware choices out there - you just need to ask the right questions to get the one that 's best for you.

Tags: iss102 | storage | cpu | 4k | gpu | codec | exabytes | Larry Jordan#
Contributing Author Larry Jordan#

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • Facilis Technology Shared Storage at IBC 2018

    Facilis Technology Shared Storage at IBC 2018

  • All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

    All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

  • Object based Storage Solution from Object Matrix at IBC 2017

    Object based Storage Solution from Object Matrix at IBC 2017

  • Glyph External Storage including the Studio and Atom raid at IBC 2017

    Glyph External Storage including the Studio and Atom raid at IBC 2017

  • Object Based Storage Solutions from Object Matrix at NAB 2017

    Object Based Storage Solutions from Object Matrix at NAB 2017

  • Storage DNA at BVE 2016

    Storage DNA at BVE 2016

  • Fibrenetix with StorageDNA at IBC 2014

    Fibrenetix with StorageDNA at IBC 2014

  • Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

    Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

  • Facilis at IBC 2013

    Facilis at IBC 2013

  • Facilis at NAB 2013

    Facilis at NAB 2013

  • New CEO and news update from TMD at NAB 2017

    New CEO and news update from TMD at NAB 2017

  • SGL Broadcast at IBC 2015

    SGL Broadcast at IBC 2015

  • Tiger Technology at BVE 2015

    Tiger Technology at BVE 2015

  • Fibrenetix with Quadrus at IBC 2014

    Fibrenetix with Quadrus at IBC 2014

  • Facilis at NAB 2014

    Facilis at NAB 2014

  • GB Labs Space at NAB 2014

    GB Labs Space at NAB 2014

  • ERA at BVE 2014

    ERA at BVE 2014

  • ERA Avere at BVE 2014

    ERA Avere at BVE 2014

  • Front Porch Digital on BroadcastShow LIVE at IBC 2013

    Front Porch Digital on BroadcastShow LIVE at IBC 2013

  • Sonnet Technologies on BroadcastShow LIVE at IBC 2013

    Sonnet Technologies on BroadcastShow LIVE at IBC 2013

  • EditShare on BroadcastShow LIVE at IBC 2013

    EditShare on BroadcastShow LIVE at IBC 2013

  • Aframe Cloud Video at IBC 2013

    Aframe Cloud Video at IBC 2013

  • NOA Audio Solutions at IBC 2013

    NOA Audio Solutions at IBC 2013

  • Global Distribution with mLogic at IBC 2013

    Global Distribution with mLogic at IBC 2013

  • SGL at NAB 2013

    SGL at NAB 2013

  • Object Matrix at BVE 2013

    Object Matrix at BVE 2013

  • Facilis Technology at BVE 2013

    Facilis Technology at BVE 2013

  • Editshare at NAB 2012

    Editshare at NAB 2012

  • Nexsan at NAB 2012

    Nexsan at NAB 2012

  • Arkivum at BVE 2012

    Arkivum at BVE 2012

  • Facilis at BVE 2012

    Facilis at BVE 2012

  • Real Life Kit at ProVideo2011

    Real Life Kit at ProVideo2011

  • Suitcase TV at IBC2011

    Suitcase TV at IBC2011

  • Sonnet Technology at IBC2011

    Sonnet Technology at IBC2011

  • ATTO Technology at IBC2011

    ATTO Technology at IBC2011

  • Matrox Unveils Monarch EDGE 4K Contribution Encoder at IBC 2018

    Matrox Unveils Monarch EDGE 4K Contribution Encoder at IBC 2018

  • 12G 4K microMUSA Jackfield and Fibre Termination Solutions from Canford at IBC 2018

    12G 4K microMUSA Jackfield and Fibre Termination Solutions from Canford at IBC 2018

  • Blackmagic Design Pocket Cinema Camera BMPCC 4K at IBC 2018

    Blackmagic Design Pocket Cinema Camera BMPCC 4K at IBC 2018

  • Apantac 4k UHD HDMI/DVI Multiviewer with KVM at IBC 2018

    Apantac 4k UHD HDMI/DVI Multiviewer with KVM at IBC 2018

  • JVC 4K Connected Camera GY-HC550 and 500 at IBC 2018

    JVC 4K Connected Camera GY-HC550 and 500 at IBC 2018

  • IMT Vislink HCAM 4k UHD transmitter at IBC 2018

    IMT Vislink HCAM 4k UHD transmitter at IBC 2018

  • Blackmagic Pocket Cinema Camera 4K at NAB 2018

    Blackmagic Pocket Cinema Camera 4K at NAB 2018

  • Blackmagic ATEM Television Studio Pro 4K at NAB 2018

    Blackmagic ATEM Television Studio Pro 4K at NAB 2018

  • 4K UHD Multi-viewer HDMI 2.0 from Apantac at NAB 2018

    4K UHD Multi-viewer HDMI 2.0 from Apantac at NAB 2018

  • Omnitek Updates for Ultra 4k Toolbox at NAB 2018

    Omnitek Updates for Ultra 4k Toolbox at NAB 2018

  • 4k and HEVC with Garland at BVE 2018

    4k and HEVC with Garland at BVE 2018

  • 6K Sony Venice and Sony 4k Palm Camcorders at IBC 2017

    6K Sony Venice and Sony 4k Palm Camcorders at IBC 2017

  • JVC 4K UHD Monitors at IBC 2017

    JVC 4K UHD Monitors at IBC 2017

  • The CFU5 4k Transmitter from Vislink at IBC 2017

    The CFU5 4k Transmitter from Vislink at IBC 2017

  • Wireless 4K Transmitter from DTC Domo Broadcast at IBC 2017

    Wireless 4K Transmitter from DTC Domo Broadcast at IBC 2017

  • AJA Thunderbolt 3 IO 4k Plus at IBC 2017

    AJA Thunderbolt 3 IO 4k Plus at IBC 2017

  • HDR, 12G, 4k, IP - the range expands from Phabrix at NAB 2017

    HDR, 12G, 4k, IP - the range expands from Phabrix at NAB 2017

  • Quad 4k Capture and Stream with Maevex from Matrox at NAB 2017

    Quad 4k Capture and Stream with Maevex from Matrox at NAB 2017

  • 4k 60 Matrix from GDSYS at NAB 2017

    4k 60 Matrix from GDSYS at NAB 2017

  • HEVC 4k Encoding from Aviwest at NAB 2017

    HEVC 4k Encoding from Aviwest at NAB 2017

  • Apantac 4k live demo at NAB 2016

    Apantac 4k live demo at NAB 2016

  • Omnitek Ultra 4K Tool Box at IBC 2015

    Omnitek Ultra 4K Tool Box at IBC 2015

  • Snell Advanced Media (SAM) 4k workflow at IBC 2015

    Snell Advanced Media (SAM) 4k workflow at IBC 2015

  • MULTIDYNE Silverback 4k at NAB 2015

    MULTIDYNE Silverback 4k at NAB 2015

  • Atomos talking 4K on BroadcastShow LIVE at IBC 2013

    Atomos talking 4K on BroadcastShow LIVE at IBC 2013

  • Panasonic Lumix GH5 at BVE 2017

    Panasonic Lumix GH5 at BVE 2017

  • Garland at BVE 2017

    Garland at BVE 2017

  • Ikegami 8k camera at IBC 2016

    Ikegami 8k camera at IBC 2016

  • Ikegami HQLM-3120W monitor at IBC 2016

    Ikegami HQLM-3120W monitor at IBC 2016

  • Ikegami UHK-430 camera at IBC 2016

    Ikegami UHK-430 camera at IBC 2016

  • Video Devices PIX-E at IBC 2016

    Video Devices PIX-E at IBC 2016

  • Sound Devices Wingman app at IBC 2016

    Sound Devices Wingman app at IBC 2016

  • JVC GY-LS300 at IBC 2015

    JVC GY-LS300 at IBC 2015

  • AVIWEST at IBC 2015

    AVIWEST at IBC 2015

  • BBright at IBC 2015

    BBright at IBC 2015

  • Thomson Video Networks at IBC 2015

    Thomson Video Networks at IBC 2015

  • Lynx Technik at IBC 2015

    Lynx Technik at IBC 2015

  • JVC at BVE 2015

    JVC at BVE 2015

  • Grass Valley at BVE 2015

    Grass Valley at BVE 2015

  • Gearhouse Broadcast at BVE 2015

    Gearhouse Broadcast at BVE 2015

  • KITPLUS rig setup at IBC 2014

    KITPLUS rig setup at IBC 2014

  • Tektronix at IBC 2014

    Tektronix at IBC 2014

  • Cooke Optics technology at IBC 2014

    Cooke Optics technology at IBC 2014

  • Cooke Optics talk anamorphic at IBC 2014

    Cooke Optics talk anamorphic at IBC 2014

  • JVC GY-LS300 prototype at IBC 2014

    JVC GY-LS300 prototype at IBC 2014

  • MultiDyne SilverBACK at IBC 2014

    MultiDyne SilverBACK at IBC 2014

  • BBright at IBC 2014

    BBright at IBC 2014

  • Hitachi at IBC 2014

    Hitachi at IBC 2014

  • Guntermann and Drunck DP-HR range at IBC 2014

    Guntermann and Drunck DP-HR range at IBC 2014

  • Ateme at IBC 2014

    Ateme at IBC 2014

  • Quantel LiveTouch at IBC 2014

    Quantel LiveTouch at IBC 2014

  • Quantel deal with AFP at IBC 2014

    Quantel deal with AFP at IBC 2014

  • Snell Kahuna Production Switcher at IBC 2014

    Snell Kahuna Production Switcher at IBC 2014

  • Matrox Monarch HD at NAB 2014

    Matrox Monarch HD at NAB 2014

  • Elemental Technologies Software-Defined Video at NAB 2014

    Elemental Technologies Software-Defined Video at NAB 2014

  • Elemental Technologies HEVC solutions at NAB 2014

    Elemental Technologies HEVC solutions at NAB 2014

  • Elemental Technologies Multiscreen Solutions at NAB 2014

    Elemental Technologies Multiscreen Solutions at NAB 2014

  • Guntermann and Drunck at NAB 2014

    Guntermann and Drunck at NAB 2014

  • Matrox Mojito at NAB 2014

    Matrox Mojito at NAB 2014

  • Matrox Capture Card at NAB 2014

    Matrox Capture Card at NAB 2014

  • Blackmagic URSA at NAB 2014

    Blackmagic URSA at NAB 2014

  • Blackmagic Studio Camera at NAB 2014

    Blackmagic Studio Camera at NAB 2014

  • Cintel Film Scanner: Blackmagic Design at NAB 2014

    Cintel Film Scanner: Blackmagic Design at NAB 2014

  • Guntermann and Drunck at BVE 2014

    Guntermann and Drunck at BVE 2014

  • Philip Bloom on Tour in Belfast

    Philip Bloom on Tour in Belfast

  • Cinegy on BroadcastShow LIVE at IBC 2013

    Cinegy on BroadcastShow LIVE at IBC 2013

  • Blackmagic Design on BroadcastShow LIVE at IBC 2013

    Blackmagic Design on BroadcastShow LIVE at IBC 2013

  • Miranda on BroadcastShow LIVE at IBC 2013

    Miranda on BroadcastShow LIVE at IBC 2013

  • Snell on BroadcastShow LIVE at IBC 2013

    Snell on BroadcastShow LIVE at IBC 2013

  • JVC GY-HM600 series camera updates at IBC 2013

    JVC GY-HM600 series camera updates at IBC 2013

  • ATEME at IBC 2013

    ATEME at IBC 2013

  • Multidyne SilverBack, BullDog and Spider at IBC 2013

    Multidyne SilverBack, BullDog and Spider at IBC 2013

  • Cinegy at IBC 2013

    Cinegy at IBC 2013

  • Blackmagic Design with the new ATEM Production Suite at IBC 2013

    Blackmagic Design with the new ATEM Production Suite at IBC 2013

  • MultiDyne: SilverBACK at NAB 2013

    MultiDyne: SilverBACK at NAB 2013

  • Quantel Pablo Rio at NAB 2013

    Quantel Pablo Rio at NAB 2013

  • Blackmagic Design: Products at NAB 2013

    Blackmagic Design: Products at NAB 2013

  • AJA at BVE 2013

    AJA at BVE 2013

  • JVC at BVE 2012

    JVC at BVE 2012

  • External GPU Enclosures from Sonnet at IBC 2017

    External GPU Enclosures from Sonnet at IBC 2017

  • Elemental Technologies at IBC2011

    Elemental Technologies at IBC2011

  • Cinegy Daniel 2 Codec shown at IBC 2018

    Cinegy Daniel 2 Codec shown at IBC 2018

  • Glensound Dante at IBC 2014

    Glensound Dante at IBC 2014

  • JVC GY-HM650 upgrade at NAB 2013

    JVC GY-HM650 upgrade at NAB 2013

  • ATOMOS at BVE North 2012

    ATOMOS at BVE North 2012

  • Telestream at NAB 2012

    Telestream at NAB 2012

  • Prodys at IBC2011

    Prodys at IBC2011


Related Shows
  • StorageDNA LIVE at BVE 2016

    StorageDNA LIVE at BVE 2016


Articles
What content providers need to know about OTT
Hiren Hindocha As OTT (Over-The-Top) technology has gotten more mature and established robust standards over the years, the concept of OTT monitoring is gaining popularity. With customer expectations soaring, it’s vital for OTT providers to deliver superior quality content. To deliver Quality of Experience (QoE) on par with linear TV broadcast, the entire system, starting from ingest to multi-bitrate encoding to delivery to CDN must be monitored continuously.
Tags: iss134 | ott monitoring | qos | logging | compliance | dash | atsc | cloud | Hiren Hindocha
Contributing Author Hiren Hindocha Click to read or download PDF
Giving Welsh sport a global audience
Adam Amor From the Ospreys Rugby Union team, to the Football Association of Wales, as well as national cycling, swimming and boxing coverage, Port Talbot based Buffoon Film and Media has been heavily involved in putting Welsh sports on the world stage.
Tags: iss134 | blackmagic | atem | buffoon | micro studio camera | Adam Amor
Contributing Author Adam Amor Click to read or download PDF
An Obituary to Timecode
Bruce Devlin - new A stoic and persistent character that stubbornly refused to change with the times, Timecode has finally passed on, but no-one has noticed. A long-lasting industry veteran, Timecode was brought into this world at an uncertain date in the late 1960s due to the needs of analogue tape workflows and the demand for synchronisation between audio and video devices. A joint activity between SMPTE and the EBU led to the work on Time and Control codes starting its journey to standardisation in the early 1970s.
Tags: iss134 | timecode | smpte | ebu | edit | Bruce Devlin - new
Contributing Author Bruce Devlin - new Click to read
Accelerated Workflows with eGPU
Mike Griggs From the UK’s National Trust to magazine publishers to manufacturers, digital content creator Mike Griggs has a wide and varied portfolio of clients for whom he creates 3D art, motion graphics and multimedia exhibits. A typical day might involve sampling birdsong near Virginia Woolf’s country estate or creating 3D animations for VR. To keep on top of these demands, Griggs wanted to take the full power of the GPU computing revolution on the road.
Tags: iss134 | sonnet | egpu | amd | post production | editing | Mike Griggs
Contributing Author Mike Griggs Click to read or download PDF
GoPro HERO 7 Review
Tim Bearder When I heard I was filming a nature restoration project in the pouring rain this week I was excited. WHY? No Cameraman enjoys the rain, surely but this time I was enthusiastic because I knew this would be the perfect opportunity to try out the brand new GoPro Hero 7 Black Edition.
Tags: iss134 | gopro | hero 7 | review | liberal media | Tim Bearder
Contributing Author Tim Bearder Click to read or download PDF