Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • 4k and 12G flexible routers and multiviewers for broadcast and professional AV

    4k and 12G flexible routers and multiviewers for broadcast and professional AV


Articles
The Future of Broadcast Technology
Sebastian Richter

Spotlight on Sebastian Richter, Vice President Media Systems at Rohde & Schwarz.

We are currently in the middle of a transition phase with migration to several new technologies, from the move to IP-based infrastructure and the shift from linear to video-on-demand (VOD).
The question for all of us is how long that transition phase will last; it is going to be faster for some customers then for others – national broadcasters, for example – it will be a slower process.

Tags: broadcast | 5g | 5g broadcast | rohde and schwarz | Sebastian Richter
Contributing Author Sebastian Richter Click to read
Spotlight on James Gilbert, Director of Product and Solution Management
James Gilbert

Over the next eight years we are going to be in transition, and within that there will be vastly different rates of change among content owners and media organisations. As a technology provider the onus is on us to be flexible and adaptable to meet this wide range of requirements from our customers.

Tags: | James Gilbert
Contributing Author James Gilbert Click to read
Spotlight on Karl Mehring, Director of Professional Services, Broadcast, Amplifier and Media
Karl Mehring

How has the role of Professional Services evolved in recent years and what vision do you have of the broadcast technology business? Covering new opportunities that the move to remote brings, new technologies such as 5G broadcast & the impact on the broadcast industry, and the challenges for broadcasters and how can they overcome them.

Tags: COTS | cloud | remote production | distribution | 5g broadcast | Karl Mehring
Contributing Author Karl Mehring Click to read
The Future of Broadcast Technology
Manfred Reitmeier

Now that OTT and VOD have become more mainstream, many commentators talk about traditional broadcast methods, like terrestrial transmission, being a thing of the past. With so many new platforms and non-traditional content services carving out a growing slice of the market, you can be forgiven for thinking that linear over-the-air television is on its way out. The reality is that the industry must strike a balance between meeting consumers’ shifting habits and the business and operational needs of content providers.

Tags: Rohde Schwarz | 5g broadcast | Manfred Reitmeier
Contributing Author Manfred Reitmeier Click to read
A switch in time: how KVM can unlock the future of broadcasting
Chris Smeeton

One of the major changes for broadcasters during the pandemic has been the shift towards remote production; by no means a new phenomenon in an IP environment, yet accelerated under lockdown to accommodate travel and gathering restrictions. A 2021 report found that almost 40% of broadcast professionals now employ remote production, up 9% on the previous year.

Tags: KVM | ARGOSY | GDSYS | KVM Tech | Chris Smeeton
Contributing Author Chris Smeeton Click to read