Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • KitPlus TV featuring mini remote heads, new ATMOS facility and choosing the right camera for you.

    KitPlus TV featuring mini remote heads, new ATMOS facility and choosing the right camera for you.


Articles
Connect A Broadcast-Quality Miniature Camera To Your Computer!
Spencer Doran Upgrade any home working or live streaming setup with a broadcast-quality professional webcam
Tags: webcam | marshall | CV503 | USB | ZOOM | Streaming | Spencer Doran
Contributing Author Spencer Doran Click to read
CLASS Is Remote Cloud Working the New Norm
Bruce Devlin - new The Tuk-tuk driver has just dropped us off at the hotel in Jaipur. The guard has let us through the gates, past the dogs and the obligatory cow rummaging in the garbage. We wash our hands, head for the bar and before the first Kingfisher beer arrives, I am connected to the virtual Mr MXF compute kingdom where a million unread emails, Slack messages, requests for help with broken workflows and rejected files are waiting on a collection of virtual servers somewhere in cloud-land.
Tags: iss142 | mrmxf | mr mxf | covid-19 | remote | gotomeeting | Bruce Devlin - new
Contributing Author Bruce Devlin - new Click to read or download PDF
State of the Nation and Virtual Social Distancing
Dick Hobbs - new Under normal circumstances, I usually make my plans for NAB in November time. It’s when the airlines tend to have sales. This year, for personal reasons, I did not get around to booking back then, and it was only at the end of February that I could start thinking about it.
Tags: iss142 | nab | kitplus show | iabm | darren whitehead | warner media | streaming | Dick Hobbs - new
Contributing Author Dick Hobbs - new Click to read or download PDF
TVFutures My Dive in to Learning
Lucy Crow In 2014 I had my first taste of underwater diving. I was on a family holiday and I happened across the experience of a lifetime. Within the first few minutes of that adventure I had decided that I wanted to be part of the diving world going forward. After countless dives and extensive training to become a scuba diving instructor, I made the decision to attend university. I knew that whatever I was going to study, I wanted it to involve my passion for Scuba.
Tags: iss142 | ccitv | university portsmouth | scuba | go-pro | blue planet | Lucy Crow
Contributing Author Lucy Crow Click to read or download PDF
The Importance of Audio Production Systems
Tom Knowles Large-scale entertainment programming presents a unique set of challenges for broadcasters. Often extremely high-profile keystones of a broadcaster’s schedule and increasingly billed as live ‘event television’, as well as delivering the very best audio quality they demand a huge degree of flexibility and agility in terms of their production. With music being such a core component of their appeal to growing audiences, in both linear broadcast and online, nowhere is that more important than their audio production systems.
Tags: iss142 | ssl | solid state logic | system T | tempest engine | binaural encoder | Tom Knowles
Contributing Author Tom Knowles Click to read or download PDF