Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • 4k and 12G flexible routers and multiviewers for broadcast and professional AV

    4k and 12G flexible routers and multiviewers for broadcast and professional AV


Articles
Sennheiser MKE 400 hands on review and test
KitPlus

Sennheiser have just released two products aimed at simplifying audio on the move, the MKE400 shotgun microphone and the XS Lav Mic, in this review we’re looking at the MKE400.

Tags: sennheiser mke400 | sennheiser mke 400 | sennheiser mke 400 quality | best microphones for youtube | mic for youtube videos | sennheiser mke 400 hands on | microphone review | iphone videography | microphones | sound for video | camera microphone | microphone for iphone | microphone for youtube | video microphone | shotgun mics | smartphone microphone | vlogging mic | best microphone for video | shotgun microphone review | mke 400 | sennheiser mke 400 hands on review | sennheiser mke 400 test | KitPlus
Contributing Author KitPlus Click to read
A Broadcasters Guide to Microservices
Roger Persson

The word “microservices” has been creeping into the conversation about software-centric systems recently. Is it really a different approach, what does it mean, and what are the advantages?

Tags: nxt edition | microservices | modular software | Roger Persson
Contributing Author Roger Persson Click to read
Streaming Technologies Explained
Bruce Devlin - new

The word streaming can relate to different technologies from different manufacturers in different scenarios and in this short article we are going to take a little tour of what it all means. 

Tags: mr mxf | streaming | hls | dash | cmaf | mpeg | Bruce Devlin - new
Contributing Author Bruce Devlin - new Click to read
REVIEW - Hollyland MARS 400S PRO Digital Wireless Transmission System
Phil Vinter

Built by Hollyland and costing around £600 the Mars 400S is an easy-to-use wireless video transmitter/receiver system – it will be right up the alley of anyone who, like me, considers an instruction manual to be nothing more than box padding.

Read Phils Full review here, and link to the video review.

Tags: HOLLYLAND | MARS 400 | wireless | tx | rx | Phil Vinter
Contributing Author Phil Vinter Click to read
HOLLYLAND LARK 150 Review
Phil Vinter

The exponential growth of video supercharged a technological revolution that began in the early noughties with the advent of DSLR cameras.
The cinematic-style shallow depth of field image video quality from DSLRs was a revelation, but, back then audio was the big drawback. As demand has grown so the technology has improved and today’s one-man-band small rig set ups can capture pretty good sound into a DSLR style camera.

Phil Vinter reviews the latest release from HOLLYLAND and it's a thumbs up from us.

Tags: HOLLYLAND | radio mic | audio | LARK-150 | CVP | Phil Vinter
Contributing Author Phil Vinter Click to read