How to Do Objective Video Testing

Over recent decades, the role of video images has grown steadily. Advances in technologies underlying the capture, transfer, storage, and display of images have created situations where communicating using images has become economically feasible. More importantly, video images are in many situations an extremely efficient way of communicating as witnessed by the proverb "a picture is worth a 1000 words."

Notwithstanding these technological advances, the current state of the art requires many compromises. Examples of these compromises are temporal resolution versus noise, spatial resolution versus image size, and luminance/color range versus gamut. These choices affect the video quality of the reproduced images. To make optimal choices, it is necessary to have knowledge about how particular choices affect the impression of the viewer. This is the central question of all video quality research.

Current video quality research can be divided into 2 approaches: experimental evaluation and modeling.

Experimental Evaluation
A group of human subjects is invited to judge the quality of video sequences under defined conditions. Several recommendations are found in ITU-R BT.500.10 "Methodology for Subjective Assessment of the quality of Television Pictures" and ITU-T P.9210 "Subjective Video Quality Assessment methods for Multimedia Applications.

The main subjective quality methods are Degradation Category Rating (DCR), Pair Comparison (PC) and Absolute Category Rating (ACR). The human subjects are shown 2 sequences (original and processed) and are asked to assess the overall quality of the processed sequence with respect to the original (reference) sequence. The test is divided into multiple sessions and each session should not last more than 30 minutes. For every session, several dummy sequences are added, which are used to train the human subjects and are not included in the final score. The subjects score the processed video sequence on a scale (usually 5 or 9) corresponding to their mental measure of the quality - this is termed Mean Observer Score (MOS).

Two serious drawbacks of this approach are:
* It is extremely time consuming, and tiresome for the participants.
* The obtained knowledge cannot be generalized because relationships between design choices and video quality are descriptive rather than based on understanding.

As a result, in a single series of experiments only a small fraction of the possible design decisions can be investigated. This makes the process even longer and more tedious.

Modeling
The second approach tries to address these drawbacks by means of developing models that describe the influences of several physical image characteristics on video quality, usually through a set of video attributes thought to determine video quality. When the influence of a set of design choices on physical video characteristics is known, then models can predict video quality. The models express video quality in terms of visible distortions, or artifacts introduced during the design process. Examples of typical distortions include flickering, blockiness, noisiness, or color shifts.

Two types of models exist, where the fundamental difference between them is how the impairment is calculated.

In the first type, physiologically or psychophysically models of early visual processing are used to calculate impairment from a difference between the video sequences. Many well known metrics exist, which compare the "original" to the "processed" output:
* PSNR - Peak Signal to Noise Ratio
* JND - Just noticeable differences
* SSIM - Structural SIMilarity
* VQM - Video Quality Metric
* MPQM - Moving Picture Quality Metric
* NVFM - Normalize Video Fidelity Metric

The two most important drawbacks of this approach are
* It is unclear what exactly the "original" version of a video is.
* These algorithms are measuring visible differences not video quality.

The second type of model tries to estimate visible distortions directly from the "processed" video; instead of comparing it to the "original". In this type of model, visible distortions of a video, such as unsharpness or noisiness are predicted by estimating physical attributes of the video. The advantage of this approach is that the "original" video sequence is not needed. The uncertain translation from visible distortions to video quality is an important drawback to this approach.

Video Clarity Solution
Regardless of whether experimentation or modeling is chosen, certain features are common:
* Video must be presented in an "unaltered" state.
* Results must be tabulated and preserved.

Since processing video may take many forms: compression, video enhancements, and color space conversions to name a few, a system must be put in place to normalize the video information so that a comparison can be done. It must be remembered that the end consumer does not care if the original video was compressed with Windows Media, DviX, or MPEG-4. All that matters is whether the video left them with the desired impression.

To streamline the process, equipment for video quality testing needs to be defined, which can capture, play, and analyze multiple video sequences. Further, as new input/output interfaces are continuously under development, the test equipment should use an open-architecture approach to ease upgradeability.

Video Clarity defined the ClearView product line with these objectives in mind.
* Capture video sequences in as many formats as possible.
* Matt/Crop all video sequences to user-selectable resolution.
* Translate all video sequences to uncompressed Y'CbCr 4:2:2 or RGB 4:4:4.
* Support 8 and 10-bit data paths with upgradeability to future 12-bit modes.
* Store the video sequences as frames (fields) so that they can be played at any rate.
* Display the video sequences in real time in multiple viewing modes.
* Playback controls include play, shuttle, jog, pause, pan, and zoom.
* Apply objective metrics to the video sequences.
* Export pieces of video sequences to further analyze off-line.
* Use a standard operating system so that the operator can run 3rd party analysis applications.

By working in the uncompressed domain, any two video processing algorithms can be compared independent of compression or other processing.

To further simplify the work flow, any video sequence can be played; while capturing another video sequence, thus, combining the video server and capture device into one unit. By doing this, the original source is already inside the test equipment for easy comparison.

The original and processed video sequences can be displayed - side-by-side, mirrored, or seamless split - on a single display. This eliminates the need to calibrate two separate displays.

ClearView applies various objective metrics to the video sequences, generates graphs, and calculates an objective score. ClearView includes PSNR, Spatial Information and Temporal Information (as proposed in ITU-T P.9210). These metrics are the basis for more sophisticate metrics like VQM, JND, which are being analyzed by organizations like VQEG, VPQM, and SMPTE.

ClearView displays multiple video sequences, even if they are played at different rates (i.e. mobile phone video compared to TV) for the expert viewers; while recording the objective metric scores. While the MOS cannot be repeated, the objective metric can, easily and readily.

Benefits
* Repeatable tests, quantitative results, and a streamlined setup.
* Analyze 2 video sequences in real-time up to 1080P.
* Input virtually any file type or capture from any digital or analog source.
* Multiple viewing modes are presented on a single display - no need to calibrate 2 separate Television displays to compare video sequences.
* Integrated uncompressed, high definition Video Server and Capture Device.
* Ability to Play 2 fully uncompressed, HD Streams in Real-Time.
* Hybrid Solution with Integrated Objective Metrics and Subjective Viewing Modes.

Implementation
ClearView takes advantage of the high-reliability of today's off-the-shelf, high-performance computer platforms. This ensures that products are made with state-of-the-art hardware, while at the same time avoiding the high cost of custom designs.

ClearView provides broadcasters, researchers, and compression developers with the unique ability to capture, play-out, and analyze video. Objective measurements are generated, graphed, and logged for repeatable tests.