Full reference testing helps ensure quality audio delivery


Adam Schadle TV-Bay Magazine
Read ezine online
by Adam Schadle
Issue 101 - May 2015

With the ever-increasing amount and variety of programming, devices, screen sizes, quality levels, and delivery methods today, broadcasters and delivery networks must be able to measure the quality of audiovisual content - and their processes for delivering it - to ensure it will meet standards that satisfy audiences, subscribers, advertisers, and regulators. One of the most accurate methods for assessing changes in quality is full-reference testing, which compares original content to processed content. Full-reference testing produces very accurate and highly repeatable results at a much lower cost than subjective testing with human listeners. Astute content distributors recognize that fact and are using full-reference testing to help them determine the best combination of equipment and delivery schemes to use in their operations.

In terms of selecting an appropriate delivery bit rate, the workflow is fairly straightforward (Figure 1): An A/V source player feeds an uncompressed signal to the encoder's input. The compressed output of the encoder can be fed directly to the decoder's input and captured in real time to form one element in a set of full-reference test sequences. Additional outputs are then captured and added to the sequence set by making small-step changes in the encoder bit rate. You can create more full-reference test sequence sets by using different source content and repeating the process of capturing and storing the decoder output sequences. (This workflow can also be performed as a file process in place of real-time play/record functions.)

Once the full-reference comparisons are made for various bit rates, differences in audio quality caused by each bit-rate change can be measured in each sequence set using objective perceptual techniques such as PEAQ (Perceptual Evaluation of Audio Quality). The PEAQ model, an international standard for audio quality measurement, was calibrated to deliver scores that correlate with the mean opinion scores of a panel of expert listeners who took part in a formal subjective test. In this sense, PEAQ can be viewed as an average expert listener score. By plotting the scores for each set of sequences on a graph, it's easy to see the quality trend for each bit rate under comparison.

PEAQ is optimized to measure the perceptual quality of wideband (20 kHz) audio codecs and has been successfully used to measure the quality of other wideband audio devices such as analog-to-digital converters, digital-to-analog converters, and sample-rate converters.

The mean quality rating is either scored as a 0 to -4 or 1 to 5. The score is an objective differential grade (ODG), which represents the overall severity of the impairments in the test signal. Its meaning is defined on the rating scales of Figure 2.

This objective, full-reference test measurement will provide a quality value for the corresponding audio in each frame of video. These frame-by-frame quality values can be compared directly to the quality values from other elements of the same test sequence set (since they all used the same source sequence), or they can be compared to other sets that have been generated using the same source sequence at another bit rate. If objective measurements vary greatly from one bit rate to another in portions of the output sequences, then you can examine them more extensively using subjective techniques.

The full-reference test and measurement process for evaluating codec technologies is very similar to the one for choosing a bit rate, except that you change the codec engine for each test instead of the bit rate (Figure 3). After evaluating the first codec (A), simply insert a second codec (B) into the test workflow and start again. From there PEAQ measurements can be made to identify trends in the average sequence scores as a result of the different codecs.

By using full-reference testing and the PEAQ audio quality scoring model, content originators can get a clear idea of how close a processed audio signal is to the original, and how it is affected by changes in different delivery variables. The results of these tests help them to ensure a highly accurate measurement of quality for audio delivery.


Tags: iss101 | audio delivery | full reference | workflow | PEAQ | bit rates | Adam Schadle
Contributing Author Adam Schadle

Read this article in the tv-bay digital magazine
Article Copyright tv-bay limited. All trademarks recognised.
Reproduction of the content strictly prohibited without written consent.

Related Interviews
  • ChyronHego show their PRIME Workflow at IBC 2018

    ChyronHego show their PRIME Workflow at IBC 2018

  • All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

    All IP Workflow from Quantum with Xcellis Workflow Storage solutions at NAB 2018

  • Object Matrix Hybrid Workflow and Artifical Intelligence at NAB 2018

    Object Matrix Hybrid Workflow and Artifical Intelligence at NAB 2018

  • Workflow Solutions from Pronology at NAB 2017

    Workflow Solutions from Pronology at NAB 2017

  • Snell Advanced Media (SAM) 4k workflow at IBC 2015

    Snell Advanced Media (SAM) 4k workflow at IBC 2015

  • LIVEU Workflow solutions at NAB 2015

    LIVEU Workflow solutions at NAB 2015

  • TMD talk workflow on BroadcastShow LIVE at IBC 2013

    TMD talk workflow on BroadcastShow LIVE at IBC 2013

  • Quantels Stream-based Workflows at IBC 2013

    Quantels Stream-based Workflows at IBC 2013

  • Streaming Media with Interra Systems at NAB 2017

    Streaming Media with Interra Systems at NAB 2017

  • New CEO and news update from TMD at NAB 2017

    New CEO and news update from TMD at NAB 2017

  • SGL Broadcast at IBC 2015

    SGL Broadcast at IBC 2015

  • Tiger Technology at BVE 2015

    Tiger Technology at BVE 2015

  • ERA - Adobe Anywhere - at BVE 2015

    ERA - Adobe Anywhere - at BVE 2015

  • NETIA at BVE 2015

    NETIA at BVE 2015

  • Dalet at IBC 2014

    Dalet at IBC 2014

  • Pronology at IBC 2014

    Pronology at IBC 2014

  • Quantel LiveTouch at IBC 2014

    Quantel LiveTouch at IBC 2014

  • Quantel deal with AFP at IBC 2014

    Quantel deal with AFP at IBC 2014

  • Snell Kahuna Production Switcher at IBC 2014

    Snell Kahuna Production Switcher at IBC 2014

  • Telestream Wirecast and Switch at IBC 2014

    Telestream Wirecast and Switch at IBC 2014

  • Telestream Vantage support for DPP at IBC 2014

    Telestream Vantage support for DPP at IBC 2014

  • Broadstream Solutions at NAB 2014

    Broadstream Solutions at NAB 2014

  • Telestream Enterprise at NAB 2014

    Telestream Enterprise at NAB 2014

  • Telestream Switch at NAB 2014

    Telestream Switch at NAB 2014

  • Dalet at NAB 2014

    Dalet at NAB 2014

  • ERA at BVE 2014

    ERA at BVE 2014

  • ERA Avere at BVE 2014

    ERA Avere at BVE 2014

  • Pixel Power on BroadcastShow LIVE at IBC 2013

    Pixel Power on BroadcastShow LIVE at IBC 2013

  • Atomos talking 4K on BroadcastShow LIVE at IBC 2013

    Atomos talking 4K on BroadcastShow LIVE at IBC 2013

  • Metus on BroadcastShow LIVE at IBC 2013

    Metus on BroadcastShow LIVE at IBC 2013

  • Volicon on BroadcastShow LIVE at IBC 2013

    Volicon on BroadcastShow LIVE at IBC 2013

  • Broadstream Solutions on the Oasys stand at IBC 2013

    Broadstream Solutions on the Oasys stand at IBC 2013

  • Oasys Chameleon at IBC 2013

    Oasys Chameleon at IBC 2013

  • Loft London Solutions with Cubix on the Oasys stand at IBC 2013

    Loft London Solutions with Cubix on the Oasys stand at IBC 2013

  • Tony Taylor from TMD talks about LTFS at IBC 2013

    Tony Taylor from TMD talks about LTFS at IBC 2013

  • Tony Taylor from TMD talks about Mediaflex CI

    Tony Taylor from TMD talks about Mediaflex CI

  • Tony Taylor from TMD talks about Post Production

    Tony Taylor from TMD talks about Post Production

  • Facilis at IBC 2013

    Facilis at IBC 2013

  • Telestream with Wirecast version five at IBC 2013

    Telestream with Wirecast version five at IBC 2013

  • Telestream with Post Producer at IBC 2013

    Telestream with Post Producer at IBC 2013

  • Yospace: Advert Insertion at NAB 2013

    Yospace: Advert Insertion at NAB 2013

  • Vimond Control Center at NAB 2013

    Vimond Control Center at NAB 2013

  • Telestream Vantage v5 at NAB 2013

    Telestream Vantage v5 at NAB 2013

  • TMD at NAB 2013: MediaFlex Reporting Module

    TMD at NAB 2013: MediaFlex Reporting Module

  • TMD at NAB 2013: MediaFlex Systems

    TMD at NAB 2013: MediaFlex Systems

  • TMD at NAB 2013: Content Intelligence

    TMD at NAB 2013: Content Intelligence

  • Telestream at BVE 2013

    Telestream at BVE 2013

  • Guntermann and Drunck GmbH at BVE 2013

    Guntermann and Drunck GmbH at BVE 2013

  • Tedial at NAB 2012

    Tedial at NAB 2012

  • Marquis Broadcast at NAB 2012

    Marquis Broadcast at NAB 2012

  • Root6 at NAB 2012

    Root6 at NAB 2012

  • Editshare at NAB 2012

    Editshare at NAB 2012

  • TMD at NAB 2012

    TMD at NAB 2012

  • PlayBox Technology at BVE 2012

    PlayBox Technology at BVE 2012

  • Suitcase TV at IBC2011

    Suitcase TV at IBC2011

  • Net Insight at IBC2011

    Net Insight at IBC2011

  • Tedial at IBC2011

    Tedial at IBC2011

  • IPV at IBC2011

    IPV at IBC2011

  • TSL at IBC2011

    TSL at IBC2011


Related Shows
  • Bubble and Squeak: Live and Loud from Monday at IBC 2013

    Bubble and Squeak: Live and Loud from Monday at IBC 2013


Articles
NAB Intelligence
Bruce Devlin - new It's that time of year again. 06:30 on road bikes on the Las Vegas Strip with a merry bunch of folks who believe they look good in lycra. As we pedal West up the hill towards Red Rock there will be stories of the year just gone and questions about what to look for at the show. I, for one, will be attending the Devoncroft event to check up on who is spending what and where. I will then look forward to the topics that might be buzzing around the halls. I predict there will be many displays of Machine Learning hiding under the banner of Artificial Intelligence. Some of these will show better picture quality, others will show personalisation engines, some will be improved search tools and yet more will be synthesising pictures and motions to replace reality with and Artificial reality that's more compelling for the viewer.
Tags: iss136 | nab | class | ai | smpte | st2110 | imf | Bruce Devlin - new
Contributing Author Bruce Devlin - new Click to read or download PDF
Remote Teams and Talent
Megan Cater If your studio works with non-local creative talent, you already know that there are opportunities and challenges associated with distributed production and post production. Bridging the distance not only allows you to find the best talent for the job anywhere in the world, it creates the potential for a diverse and globally-minded workforce that boosts the creativity and vision of your entire company.
Tags: iss136 | signiant | file acceleration | ftp | dropbox | sharepoint | slack | saas | media shuttle | Megan Cater
Contributing Author Megan Cater Click to read or download PDF
Flexible Connectivity Hits Europe
Yvonne Monterroso The low-latency Dejero CellSat network was launched in Europe in January of 2019, which is excellent news for European broadcasters who can now benefit from the same flexibility CellSat provides North American Dejero customers, such as World Racing Group (WRG).
Tags: iss136 | dejero | wrg | cellsat | ku sat | is-35e | cellular | Yvonne Monterroso
Contributing Author Yvonne Monterroso Click to read or download PDF
Keeping Your Post Prodction on Track with Subclips and Search Bins
Alex Macleod

For my 2nd Kit Plus article I thought I’d try and build on the theme of my first, and that’s one of making sure things are organised at all levels of your post production projects.

Last time I talked about trying as best as you can to stick to the ‘two week rule’, making sure that the names & locations of every asset you import, and every bin & sequence that you create in your project - will make sense to you regardless of how long it is you spend away from it.

Tags: iss136 | mediacity training | subclip | premiere pro | gvs | bve | bve2019 | Alex Macleod
Contributing Author Alex Macleod Click to read or download PDF
Living LIVE at BVE with Ross Video
Jon Pratchett KitPlus’ Jon Pratchett chats to Stuart Russell, Ross Video, in the BVE studio. Covering customer partnerships and loyalty, pain points, growth, industry trends, pace of IP adoption, 12g SDI technologies, UHD / HDR, 8k, AI and we try to get a hint of new products on the horizon...
Tags: iss136 | ross video | 12g sdi | uhd | hdr | ip | bve 2019 | bvexpo | Jon Pratchett
Contributing Author Jon Pratchett Click to read or download PDF