Live streams versus srowing files


Bruce Devlin TV-Bay Magazine
Read ezine online

So, what is the difference between a stream and a file? It may seem like a dumb and obvious question, but it's crucial in the design of many file based workflows and the applications that drive them.

Let's start with the basic and obvious answer. "You can random access a file but you sequentially process a stream". This used to be important back in the days when CPUs were slow and storage was expensive, but now that we can process UHD in software in real time and my phone can hold a visually lossless HD movie on its internal storage, this answer seems rather dated.

The reality is that a good application ought to be able to treat media like a stream when needed and like a file when needed. For example, if you're pulling a big file from offline or cloud storage, it will be arriving locally as though it were a stream. It might be going slower than real time or maybe faster than real time, but if it's a good file format then the software ought to be able to handle it like a stream. This is often called the growing file or the while scenario.

Another example might be the reception of a live IP stream in a well known codec (e.g. JPEG2000) into a facility. While it's arriving, there is an expectation of good, live stream behaviour i.e. all the synchronised elements are close to each other in the stream as well as being correctly labelled so that a device joining the stream mid-transmission is able to identify what the elements are. It would also be very nice for a device joining mid-stream to know how much of the stream it had missed. These requirements are also shared with the growing file scenario.

In fact, when you list the requirements for a good live stream and a good growing file, you find that the real differences come down to:-

Latency (time delay) between acquiring the stream and displaying / using it

Knowledge of the file's history and its future e.g. index tables

How the timing and synchronisation references are used

Let's take these one by one.

Latency. In a live streaming application, the goal is almost always to minimise this. There is always a balance between cost, complexity and latency but lower latency for live streaming is nearly always better. In a growing file scenario, there is usually more tolerance (sometimes minutes rather than seconds) for the latency of using the received data when compared to a live stream. This fact can be used in the internal structure of the file when designing the multiplex or partitions or index tables to reduce overheads and improve efficiency.

History. In a growing file scenario, the complete file may exist somewhere in the cloud or on external storage and an index table for the entire file may exist. In the MXF world, the index tables can be chunked to allow the file to be used in 10sec or 1min or 10min chunks depending on the application with the full index table being transmitted at the end of the file. In contrast, a live stream multiplex will often omit index tables and they can be recreated when the file is ingested or rewrapped on reception.

Timing & Sync. In a file, the timing and synchronisation are usually relative to the internals of the file, whereas in a live stream, the timing could be relative to the clock on the wall or to the start of transmission. In the work being carried out on the synchronisation of component IP streams, this becomes important when different elements in the stream might be sent via different routes and then re-synchronised.

Why is any of this important? It's because we are living in a world where moving data over IP networks is less of a technological barrier and the economics of the data move are rising higher in the list of things to consider. One approach for HTTP streaming uses chunks of transport stream (a live streaming multiplex) stored as files on a server and then contiguously streamed to achieve a streaming experience. Many live-highlights editing workflow use MXF files growing on disc to be randomly accessed by editing software to build a package while the file is coming in.

If you're designing a workflow where you need to manipulate the content while it's arriving, the decision of the format on the wire won't be a technological one for much longer. It will be a commercial decision based on the requirements of the workflow and the equipment available.

Is it a file? Is it a stream? Who cares! Put a blue cape on it and call it superman. It's getting the right results at the right time for the right cost that counts.


Tags: iss117 | class | streaming | mr mxf | jpeg2000 | mxf | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic

  • House of Worship Multisite Streaming from VITEC at NAB 2017

    House of Worship Multisite Streaming from VITEC at NAB 2017

  • Streaming Media with Interra Systems at NAB 2017

    Streaming Media with Interra Systems at NAB 2017

  • Live Streaming Solution from Verizon at NAB 2017

    Live Streaming Solution from Verizon at NAB 2017

  • Facebook and YouTube live integration with Monarch HD from Matrox at NAB 2017

    Facebook and YouTube live integration with Monarch HD from Matrox at NAB 2017

  • LiveMedia Server and Live Xpert from 3D Storm at NAB 2017

    LiveMedia Server and Live Xpert from 3D Storm at NAB 2017

  • VITEC EZ TV at IBC 2016

    VITEC EZ TV at IBC 2016

  • JVC at IBC 2016

    JVC at IBC 2016

  • JVC GY-LS300 at IBC 2015

    JVC GY-LS300 at IBC 2015

  • VITEC at IBC 2015

    VITEC at IBC 2015

  • Matrox at IBC 2015

    Matrox at IBC 2015

  • JVC GY-LS300 prototype at IBC 2014

    JVC GY-LS300 prototype at IBC 2014

  • Matrox at IBC 2014

    Matrox at IBC 2014

  • Telestream Wirecast and Switch at IBC 2014

    Telestream Wirecast and Switch at IBC 2014

  • Telestream Vantage support for DPP at IBC 2014

    Telestream Vantage support for DPP at IBC 2014

  • NewTek TalkShow at IBC 2014

    NewTek TalkShow at IBC 2014

  • Matrox Monarch HD at NAB 2014

    Matrox Monarch HD at NAB 2014

  • Telestream Enterprise at NAB 2014

    Telestream Enterprise at NAB 2014

  • Telestream Switch at NAB 2014

    Telestream Switch at NAB 2014

  • JVC GY-HM850 and GY-HM650 at NAB 2014

    JVC GY-HM850 and GY-HM650 at NAB 2014

  • Matrox Mojito at NAB 2014

    Matrox Mojito at NAB 2014

  • Matrox Capture Card at NAB 2014

    Matrox Capture Card at NAB 2014

  • NewTek TriCaster at BVE 2014

    NewTek TriCaster at BVE 2014

  • Matrox Monarch HD and VS4 at BVE 2014

    Matrox Monarch HD and VS4 at BVE 2014

  • JVC HM850 at BVE 2014

    JVC HM850 at BVE 2014

  • Dejero on BroadcastShow LIVE at IBC 2013

    Dejero on BroadcastShow LIVE at IBC 2013

  • Pixel Power on BroadcastShow LIVE at IBC 2013

    Pixel Power on BroadcastShow LIVE at IBC 2013

  • Canon on BroadcastShow LIVE at IBC 2013

    Canon on BroadcastShow LIVE at IBC 2013

  • Harmonics Peter Alexander on BroadcastShow LIVE at IBC 2013

    Harmonics Peter Alexander on BroadcastShow LIVE at IBC 2013

  • Haivision on BroadcastShow LIVE at IBC 2013

    Haivision on BroadcastShow LIVE at IBC 2013

  • Front Porch Digital on BroadcastShow LIVE at IBC 2013

    Front Porch Digital on BroadcastShow LIVE at IBC 2013

  • Atomos with the Samurai Blade on BroadcastShow LIVE at IBC 2013

    Atomos with the Samurai Blade on BroadcastShow LIVE at IBC 2013

  • ChyronHego on BroadcastShow LIVE at IBC 2013

    ChyronHego on BroadcastShow LIVE at IBC 2013

  • Cinegy on BroadcastShow LIVE at IBC 2013

    Cinegy on BroadcastShow LIVE at IBC 2013

  • Trilogy on BroadcastShow LIVE at IBC 2013

    Trilogy on BroadcastShow LIVE at IBC 2013

  • Projects Department on BroadcastShow LIVE at IBC

    Projects Department on BroadcastShow LIVE at IBC

  • Blackmagic Design on BroadcastShow LIVE at IBC 2013

    Blackmagic Design on BroadcastShow LIVE at IBC 2013

  • Atomos talking 4K on BroadcastShow LIVE at IBC 2013

    Atomos talking 4K on BroadcastShow LIVE at IBC 2013

  • Calrec on BroadcastShow LIVE at IBC 2013

    Calrec on BroadcastShow LIVE at IBC 2013

  • Glensound on BroadcastShow LIVE at IBC 2013

    Glensound on BroadcastShow LIVE at IBC 2013

  • Zylight on BroadcastShow LIVE at IBC 2013

    Zylight on BroadcastShow LIVE at IBC 2013

  • Nexidias Drew Lanham on BroadcastShow LIVE at IBC 2013

    Nexidias Drew Lanham on BroadcastShow LIVE at IBC 2013

  • Carl Zeiss on BroadcastShow LIVE at IBC 2013

    Carl Zeiss on BroadcastShow LIVE at IBC 2013

  • Miranda on BroadcastShow LIVE at IBC 2013

    Miranda on BroadcastShow LIVE at IBC 2013

  • Orad Hi Tec Systems on BroadcastShow LIVE at IBC 2013

    Orad Hi Tec Systems on BroadcastShow LIVE at IBC 2013

  • Photon Beard on BroadcastShow LIVE at IBC 2013

    Photon Beard on BroadcastShow LIVE at IBC 2013

  • Polecam talk accessories on BroadcastShow LIVE 2013

    Polecam talk accessories on BroadcastShow LIVE 2013

  • TMD talk workflow on BroadcastShow LIVE at IBC 2013

    TMD talk workflow on BroadcastShow LIVE at IBC 2013

  • Metus on BroadcastShow LIVE at IBC 2013

    Metus on BroadcastShow LIVE at IBC 2013

  • Guntermann and Drunck on BroadcastShow LIVE at IBC 2013

    Guntermann and Drunck on BroadcastShow LIVE at IBC 2013

  • Brainstorm on BroadcastShow LIVE at IBC 2013

    Brainstorm on BroadcastShow LIVE at IBC 2013

  • Pilat Media on BroadcastShow LIVE at IBC 2013

    Pilat Media on BroadcastShow LIVE at IBC 2013

  • Sonnet Technologies on BroadcastShow LIVE at IBC 2013

    Sonnet Technologies on BroadcastShow LIVE at IBC 2013

  • EditShare on BroadcastShow LIVE at IBC 2013

    EditShare on BroadcastShow LIVE at IBC 2013

  • Nucomm on BroadcastShow LIVE at IBC 2013

    Nucomm on BroadcastShow LIVE at IBC 2013

  • Wohler Technologies on BroadcastShow LIVE at IBC 2013

    Wohler Technologies on BroadcastShow LIVE at IBC 2013

  • Nexidia QC on BroadcastShow LIVE at IBC 2013

    Nexidia QC on BroadcastShow LIVE at IBC 2013

  • SIS LIVEs Martyn Hopkins on BroadcastShow LIVE at IBC 2013

    SIS LIVEs Martyn Hopkins on BroadcastShow LIVE at IBC 2013

  • Atomos with the Ronin on BroadcastShow LIVE at IBC 2013

    Atomos with the Ronin on BroadcastShow LIVE at IBC 2013

  • Snell on BroadcastShow LIVE at IBC 2013

    Snell on BroadcastShow LIVE at IBC 2013

  • Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

    Harmonics Tom Lattie on BroadcastShow LIVE at IBC 2013

  • Primestream on BroadcastShow LIVE at IBC 2013

    Primestream on BroadcastShow LIVE at IBC 2013

  • NewTek on BroadcastShow LIVE at IBC 2013

    NewTek on BroadcastShow LIVE at IBC 2013

  • Wyplay on BroadcastShow LIVE at IBC 2013

    Wyplay on BroadcastShow LIVE at IBC 2013

  • Volicon on BroadcastShow LIVE at IBC 2013

    Volicon on BroadcastShow LIVE at IBC 2013

  • Forbidden Technologies on BroadcastShow LIVE at IBC 2013

    Forbidden Technologies on BroadcastShow LIVE at IBC 2013

  • Atomos with the Datacolor Spyder on BroadcastShow LIVE at IBC 2013

    Atomos with the Datacolor Spyder on BroadcastShow LIVE at IBC 2013

  • Sound Devices on BroadcastShow LIVE at IBC 2013

    Sound Devices on BroadcastShow LIVE at IBC 2013

  • Audio Network on BroadcastShow LIVE at IBC 2013

    Audio Network on BroadcastShow LIVE at IBC 2013

  • Cooke Optics on BroadcastShow LIVE at IBC 2013

    Cooke Optics on BroadcastShow LIVE at IBC 2013

  • SIS LIVEs Mark Shadbolt on BroadcastShow LIVE at IBC 2013

    SIS LIVEs Mark Shadbolt on BroadcastShow LIVE at IBC 2013

  • Softron Media Services on BroadcastShow LIVE at IBC 2013

    Softron Media Services on BroadcastShow LIVE at IBC 2013

  • Cobham on BroadcastShow LIVE at IBC 2013

    Cobham on BroadcastShow LIVE at IBC 2013

  • Riedel Communications on BroadcastShow LIVE at IBC 2013

    Riedel Communications on BroadcastShow LIVE at IBC 2013

  • TMD talk asset management solutions on BroadcastShow LIVE at IBC 2013

    TMD talk asset management solutions on BroadcastShow LIVE at IBC 2013

  • SMPTE on BroadcastShow LIVE at IBC 2013

    SMPTE on BroadcastShow LIVE at IBC 2013

  • Behind the scenes of the Teradek booth on BroadcastShow LIVE at IBC 2013

    Behind the scenes of the Teradek booth on BroadcastShow LIVE at IBC 2013

  • Wowza Media Systems on BroadcastShow LIVE at IBC 2013

    Wowza Media Systems on BroadcastShow LIVE at IBC 2013

  • Stream UK on BroadcastShow LIVE at IBC 2013

    Stream UK on BroadcastShow LIVE at IBC 2013

  • Cambridge Imaging Systems on BroadcastShow LIVE at IBC 2013

    Cambridge Imaging Systems on BroadcastShow LIVE at IBC 2013

  • Digital Vision on BroadcastShow LIVE at IBC 2013

    Digital Vision on BroadcastShow LIVE at IBC 2013

  • Polecam talk high speed on BroadcastShow LIVE at IBC 2013

    Polecam talk high speed on BroadcastShow LIVE at IBC 2013

  • Haivision live encoding HEVC at IBC 2013

    Haivision live encoding HEVC at IBC 2013

  • SIS LIVE and their products ManPak and DriveForce at IBC 2013

    SIS LIVE and their products ManPak and DriveForce at IBC 2013

  • Softron Media at IBC 2013

    Softron Media at IBC 2013

  • Telestream with Wirecast version five at IBC 2013

    Telestream with Wirecast version five at IBC 2013

  • Telestream with Post Producer at IBC 2013

    Telestream with Post Producer at IBC 2013

  • Yospace: Advert Insertion at NAB 2013

    Yospace: Advert Insertion at NAB 2013

  • Vimond Control Center at NAB 2013

    Vimond Control Center at NAB 2013

  • Matrox at BVE 2013

    Matrox at BVE 2013

  • Autocue at BVE 2013

    Autocue at BVE 2013

  • StreamUK at BVE North 2012

    StreamUK at BVE North 2012

  • Radiant Grid at NAB 2012

    Radiant Grid at NAB 2012

  • Haivision at NAB 2012

    Haivision at NAB 2012

  • Broadcast Pix at NAB 2012

    Broadcast Pix at NAB 2012

  • Cinegy at NAB 2012

    Cinegy at NAB 2012

  • LiveU at NAB 2012

    LiveU at NAB 2012

  • SIS LIVE at NAB 2012

    SIS LIVE at NAB 2012

  • YoSpace at NAB 2012

    YoSpace at NAB 2012

  • Anton Bauer at NAB 2012

    Anton Bauer at NAB 2012

  • Garland Partners at BVE 2012

    Garland Partners at BVE 2012

  • StreamUK at BVE North 2011

    StreamUK at BVE North 2011

  • Suitcase TV at IBC2011

    Suitcase TV at IBC2011

  • Digital Rapids at IBC2011

    Digital Rapids at IBC2011

  • NuGen Audio at IBC 2013

    NuGen Audio at IBC 2013

  • Arkivum at BVE 2012

    Arkivum at BVE 2012


Related Shows
  • Larry Jordan with Thom Holliday talk streaming at BVE 2016

    Larry Jordan with Thom Holliday talk streaming at BVE 2016


Articles
Accelerated Workflows with eGPU
Mike Griggs From the UK’s National Trust to magazine publishers to manufacturers, digital content creator Mike Griggs has a wide and varied portfolio of clients for whom he creates 3D art, motion graphics and multimedia exhibits. A typical day might involve sampling birdsong near Virginia Woolf’s country estate or creating 3D animations for VR. To keep on top of these demands, Griggs wanted to take the full power of the GPU computing revolution on the road.
Tags: iss134 | sonnet | egpu | amd | post production | editing | Mike Griggs
Contributing Author Mike Griggs Click to read or download PDF
Giving Welsh sport a global audience
Adam Amor From the Ospreys Rugby Union team, to the Football Association of Wales, as well as national cycling, swimming and boxing coverage, Port Talbot based Buffoon Film and Media has been heavily involved in putting Welsh sports on the world stage.
Tags: iss134 | blackmagic | atem | buffoon | micro studio camera | Adam Amor
Contributing Author Adam Amor Click to read or download PDF
Keeping it remotely real
Reuben Such Everyone wants to do more with less. Always have, although it could be argued that doing more with more is something to aspire to, not many have that luxury. So let’s stick with the prevailing winds of doing more with less, and not just doing more, but doing it remotely, particularly in terms of production. Remote production, in particular, is getting a lot of attention in the field these days, but not so much in terms of the remote operation of fixed studios.
Tags: iss134 | remote control | IPE | IDS | Reuben Such
Contributing Author Reuben Such Click to read or download PDF
What content providers need to know about OTT
Hiren Hindocha As OTT (Over-The-Top) technology has gotten more mature and established robust standards over the years, the concept of OTT monitoring is gaining popularity. With customer expectations soaring, it’s vital for OTT providers to deliver superior quality content. To deliver Quality of Experience (QoE) on par with linear TV broadcast, the entire system, starting from ingest to multi-bitrate encoding to delivery to CDN must be monitored continuously.
Tags: iss134 | ott monitoring | qos | logging | compliance | dash | atsc | cloud | Hiren Hindocha
Contributing Author Hiren Hindocha Click to read or download PDF
An Obituary to Timecode
Bruce Devlin - new A stoic and persistent character that stubbornly refused to change with the times, Timecode has finally passed on, but no-one has noticed. A long-lasting industry veteran, Timecode was brought into this world at an uncertain date in the late 1960s due to the needs of analogue tape workflows and the demand for synchronisation between audio and video devices. A joint activity between SMPTE and the EBU led to the work on Time and Control codes starting its journey to standardisation in the early 1970s.
Tags: iss134 | timecode | smpte | ebu | edit | Bruce Devlin - new
Contributing Author Bruce Devlin - new Click to read