Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • 4k and 12G flexible routers and multiviewers for broadcast and professional AV

    4k and 12G flexible routers and multiviewers for broadcast and professional AV


Articles
A switch in time: how KVM can unlock the future of broadcasting
Chris Smeeton

One of the major changes for broadcasters during the pandemic has been the shift towards remote production; by no means a new phenomenon in an IP environment, yet accelerated under lockdown to accommodate travel and gathering restrictions. A 2021 report found that almost 40% of broadcast professionals now employ remote production, up 9% on the previous year.

Tags: KVM | ARGOSY | GDSYS | KVM Tech | Chris Smeeton
Contributing Author Chris Smeeton Click to read
Rohde & Schwarz integration with Adobe Premiere Pro
Ciaran Doran

Rohde & Schwarz SpycerNode storage & VENICE ingest are perfect partners for Adobe workflows

Tags: Rohde Schwarz | media | VENICE | SpycerNode | adobe | premier pro | Ciaran Doran
Contributing Author Ciaran Doran Click to read
Cloud integration: The only way is up
Francois Vaillant

In 2017 we saw the introduction of SMPTE 2110 and since then, the transition to IP has emerged as a priority for broadcasters looking to streamline their operations, decrease their footprint and effectively integrate with an industry in a rapid state of flux. A 2020 survey by Devoncroft found that almost 20% of broadcasters had already deployed SMPTE 2110, and over 25% were planning to do so. Then came the pandemic, precipitating the need for on-the-fly collaboration and accelerating the transition to remote, virtual and cloud production.

Tags: dejero | smpte 2110 | dazzl | streaming | Francois Vaillant
Contributing Author Francois Vaillant Click to read
The Cloud - a measured approach
Ciaran Doran

With the buzz of ‘cloud’ everywhere in our industry it would be natural to think that cloud is the only game in town. Isn’t it time to step back and consider very carefully how, or whether, you make that journey to the cloud?

Tags: cloud | rohde and schwarz | rohde | schwarz | Ciaran Doran
Contributing Author Ciaran Doran Click to read
Meeting the specification
Chris Smeeton

A good technical specification will detail precisely what is required, from the equipment to the cables connecting it. Many specifications will give  particular manufacturers and model numbers. On many occasions, this makes tendering simple and gives vendors a secure and fair way to bid.

Tags: CPR specification | argosy cable | fire safe cable | eu standard cable | chris smeeton | Chris Smeeton
Contributing Author Chris Smeeton Click to read