Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • BVE 2013 Day 2

    BVE 2013 Day 2


Articles
Your two week editing future
Alex Macleod

So here we are - January again! Usually a good time to reflect on the year just gone by, and a good time to look forward to the coming months as the new year begins.

When I was reflecting on my 2018, and when thinking about what to write for my first article for Kit Plus - I kept coming back to one theme - organisation.

Tags: iss135 | editing | mediacity training | premiere pro | dit | Alex Macleod
Contributing Author Alex Macleod Click to read or download PDF
Test, Measurement and Standards
Alan Wheable The Alliance for IP Media Solutions (AIMS), is a non-profit trade alliance that fosters the adoption of one set of common, ubiquitous, standards-based protocols for interoperability over IP in the media and entertainment, and professional audio/video industries.
Tags: iss135 | omnitek | aims | SNMP | hdr | ai | Alan Wheable
Contributing Author Alan Wheable Click to read or download PDF
The making of The Heist
Tom Hutchings Shine TV has never been one to shy away from a challenge, be that in terms of using new technologies, filming ideas or overall formats: we pride ourselves on being ambitious and risk-takers.
Tags: iss135 | liveu | heist | streaming | cellular | mobile | connectivity | Tom Hutchings
Contributing Author Tom Hutchings Click to read or download PDF
Grading BBC Sounds
Simone Grattarola

The BBC has launched its new personalised music, radio and podcast app with a campaign that follows one listener’s journey from meeting Kylie Minogue in a lift to Idris Elba on a bus. 

BBC Sounds offers a single home for the BBC’s thousands of hours of audio content, including live and on-demand shows and special music mixes curated by artists.

BBC Creative, the broadcaster’s in-house creative division, took the brief to agency Riff Raff Films and Megaforce directing duo of Charles Brisgand and Raphaël Rodriguez who in turn brought on board regular collaborators Time Based Arts.

Tags: iss135 | bbc | grading | bbc sounds | davinici | resolve | blackmagic | editing | Simone Grattarola
Contributing Author Simone Grattarola Click to read or download PDF
21st Century Technology for 20th Century Content
James Hall A big challenge facing owners of legacy content is rationalising and archiving their tape and film-based media in cost effective and efficient ways, whilst also adding value. Normally the result of this is to find a low cost means of digitising the content – usually leaving them with a bunch of assets on HDD. But then what? How can content owners have their cake and eat it?
Tags: iss135 | legacy | digitising | digitizing | archive | James Hall
Contributing Author James Hall Click to read or download PDF