Troubleshooting - Top Tips to Make Files Work


Bruce Devlin TV-Bay Magazine
Read ezine online
by Bruce Devlin
Issue 110 - February 2016

Troubleshooting - Top Tips to Make Files Work

So, you built yourself the world's most perfect, IT-based file workflow and you have a state-of-the-art media facility? Congratulations, have a beer, but at some point it's going to go wrong!

Given that fewer than 50% of support cases in any company that I have worked at result in changes to software, it helps to have a good analytical approach to finding the causes of problems and communicating them effectively to your suppliers.

What do you do? How do you find those problems?

The first approach is to separate symptoms from causes, and being specific is crucial. Telling support: Our MAM system triggers an API refresh cycle at 2 minutes past every hour, which seems to happen 5 minutes before the storage anomaly is a great starting point. This phrase informs support that you are seeing something happen regularly and that there is some correlation between the symptom and another event in the system / workflow.

As we all know, correlation does not imply causation. So how do you find causes and not symptoms?

One of my favourite troubleshooting techniques for the IT and the broadcast & media industry is The 5 Whys. This technique is widely attributed to one of my all-time engineering heroes Sakichi Toyoda. Wikipedia has the following definition: The 5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ˜Why?'

Here is an example where chaining the question why:

The Problem: The media file can't be transcoded

Why #1 The source file can't be opened.

Why #2 Permission failure. The file cannot be opened from the transcode machine with the account associated with the transcode service.

Why #3 Group settings. We check the account on the transcode machine and discover it is in the wrong group so we change it. It still won't open.

Why #4 Drive mount settings. The way in which the drive was mounted in the operating system had an override for the group control. Updating the mount instruction remaps the group to the correct value and the system works. But why did the system go from working to not working?

Why #5 System maintenance. Updating the security of the infrastructure is always risky. You can test 99% of the use cases, but unless you knew to look for the fact that a single machine in a farm was hand-installed in a rush and had the wrong mount overrides then you'll never find the cause of the symptoms until something else changes. Remember that we see symptoms and report them to support, we rarely directly see the causes.

Let's look now at some other questions you can ask yourself when troubleshooting your file-based workflow¦

Is the issue repeatable? Being able to reproduce the symptoms greatly helps to isolate, identify and resolve an issue.

Resolution dependent? Let's try to switch between SD, HD and UHD at different frame rates “ do the symptoms still show? Try to look at the dependency of each parameter of the input.

Is the issue environmental-related? Does the failure occur when there are more people on the system? At a particular time of day? On a specific day of the week? When the playout servers are being updated? Only on specific servers? Only on AWS and OpenShift, but not on Azure?

Time dependent? Does it only fail at a particular time of the day?

Conclusion I won't pretend that finding the causes of any issue in a big software system is easy. First and foremost, it's crucial to differentiate between symptoms and causes. It requires dedication and a methodical nature as well as extreme calm when all around you are panicking. Just remember that at the end of the day, the support person on the end of the telephone is a human who is trying to help find the causes that are hidden within the symptoms that you are seeing. Be nice to them!


Tags: iss110 | dalet | class | academy | sakichi toyoda | openshoft | azure | Bruce Devlin
Contributing Author Bruce Devlin

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • DALET ACADEMY NAB 2015

    DALET ACADEMY NAB 2015

  • Dalet at NAB 2016

    Dalet at NAB 2016

  • Dalet at IBC 2015

    Dalet at IBC 2015

  • DALET SportsPack at NAB 2015

    DALET SportsPack at NAB 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • Dalet at IBC 2013

    Dalet at IBC 2013

  • Dalet at NAB 2013

    Dalet at NAB 2013

  • Dalet at NAB 2012

    Dalet at NAB 2012

  • Classic tubes reinvented by Kino Flo at IBC 2018

    Classic tubes reinvented by Kino Flo at IBC 2018

  • Winner of the LP54 Miller Classic

    Winner of the LP54 Miller Classic


Related Shows
  • BVE 2013 Day 2

    BVE 2013 Day 2


Articles
Creating Authentic Content That Counts
Frank le Mair We’re deep into the 2010’s and the way in which we consume content has changed forever. Broadcasters and content owners are fighting for eyeballs in a saturated market where consumers are watching their favourite shows on different devices across a number of platforms. To target millennials and younger generations, who generally consume short bursts of video on YouTube and social media - particularly Snapchat, Instagram and now IGTV - media companies are creating more and more authentic stories and are using platforms that are compelling for this demographic. Unlike generations before them, they have totally new video viewing patterns and ideals.
Tags: iss133 | insight tv | monster energy | amazon | millennial | Frank le Mair
Contributing Author Frank le Mair Click to read or download PDF
Managing Technological Change
Alan Wheable Continual technological change in the broadcast and media industries can make it difficult to plan for the mid to long term. Typically, broadcasters and media organisation are still implementing the last set of changes to working practices when the next changes come along.
Tags: iss133 | omnitek | ip | waveform | vectorscope | ultra tq | Alan Wheable
Contributing Author Alan Wheable Click to read or download PDF
Smashing the WTA Tour
Danny Ridler The Women’s Tennis Association (WTA) is the global leader in women’s professional sport with more than 2500 players representing nearly 100 nations competing for a record $146 million in prize money. The 2018 WTA competitive season includes 54 events and four Grand Slams in 30 countries. In 2017, the WTA was watched around the world by a total TV audience of 500 million – with host broadcast services provided by NEP UK.
Tags: iss133 | tennis | wta | nep | outside broadcast | ob | Danny Ridler
Contributing Author Danny Ridler Click to read or download PDF
Perimeter LED screens management
Nicolas Houel Opened in January 2016, Parc Olympique Lyonnais, also known as Groupama Stadium, is the new home of Olympique Lyonnais football club, one of the most popular clubs in France. Since its inauguration, the stadium was a host of UEFA Euro 2016, and was also chosen to stage, among other important events, the 2018 UEFA Europa League Final and football at the 2024 Summer Olympics.
Tags: iss133 | 3dstorm | graphics | groupama stadium | liveexpert | livecg | deltacast | Nicolas Houel
Contributing Author Nicolas Houel Click to read or download PDF
Fast-moving productions need multi-cam RF reliability
Darren Bilton Wireless acquisition creates a level of freedom not possible with any other form of filming yet only a decade ago the technique was barely possible. The technology enabling live real-time connections over radio frequencies has advanced leaps and bounds such that there is barely an entertainment, documentary, sports or news format today that doesn’t benefit from at least one link system. That means the demands on the kit continues to rise and includes the need for fail safe multi-camera operation, ease of use and backwards compatibility all within a small, lightweight and budget-friendly package.
Tags: iss133 | boxx tv | wireless | mpeg-4 | multicam | rf | multi-cam | Darren Bilton
Contributing Author Darren Bilton Click to read or download PDF