MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Return to Subjective Comparison of Modern Video Codecs home page!

Part 1. Introduction

Contents

  • Introduction
  • Results of subjective comparison
  • Evaluation of objective metrics
  • Conclusions
  • Gratitude

    Authors want to thank Stass Soldatov and Alexey Moskvin for help in preparation and verification of the report, and all of the experts who took part in a quality assessment for their priceless help in creation of this comparison.

    Comparison abstract

    In this comparison we evaluate the quality of four popular codecs on two bitrates (690 and 1024 kbps) using subjective assessment. We use SAMVIQ as a subjective testing methodology. An average subjective opinion on the quality of an encoded video is the most important characteristic of codec's performance. Fifty experts took part in the subjective assessment; their marks form resulting codec's rating. In addition, we measured objective metrics (PSNR, VQM and SSIM) and evaluated their ability to predict subjective opinion on our test set. Results of subjective evaluation prove that x264 codec of H.264 standard provides a significantly better subjective quality than other widely-spread codecs that took part in the comparison. It is shown that PSNR cannot always be reliable measure of video quality in practical cases (i.e., not specially created to corrupt the measure).

    Overview

    Codecs

    Codec

    Vendor

    Version

    DivX

    DivX Networks

    6.0 b1571-CenterOfTheSun

    XviD

    Open Source project

    1.1.-125 ("xvid-1.1.0-beta2")

    x264

    Open Source project

    Core 48 svn-352M by Sharktooth

    WMV

    Microsoft Corporation

    9.0

    Encoders' settings

    Codec

    Parameter

    Values

    DivX

    Bitrate

    690 kbps, 1024 kbps

    XviD

    Target bitrate

    690 kbps, 1024 kbps

    x264

    Average Bitrate

    690 kbps, 1024kbps

    WMV

    Bit rate

    700000 bps, 1048576 bps

    Other settings were left to defaults. Default values are the values that are set after a codec is installed, you can see them on codecs' screenshots.

    Decoders' settings

    Decoders that are provided with codec were used for decoding of sequences of all codecs except for x264. For decoding files compressed with x264 we used popular tool "ffdshow", version of ff_x264.dll is 33 by Milan Cutca. All decoders' settings were let to defaults.

    Sequences

    The following table contains properties of the encoded sequences that were shown to experts (see "Encoding of sequences").

    Name

    Length [frames]

    Length [seconds]

    Resolution

    Source

    Battle

    257

    10.71

    704x288

    MPEG2 (DVD)

    Rancho

    240

    10.01

    704x288

    MPEG2 (DVD)

    Matrix sc.1

    250

    10.00

    720x416

    MPEG2 (DVD)

    Matrix sc.2

    250

    10.00

    720x416

    MPEG2 (DVD)


    Rules and goals of the testing

    Goals of the subjective codecs assessment

    During last few years many comparisons of video, audio and image codecs were carried out by our Graphics & Media Lab at Moscow State University (available at www.compression.ru/video). All of them used objective metrics like PSNR, VQM or SSIM. This fact has raised reasonable questions on adequacy of objective measures to the subjectively perceived quality.

    Some organizations like VQEG (Video Quality Experts Group) and ITU (International Telecommunication Union) have held subjective assessments of video quality and evaluated adequacy of objective video quality metrics [5]. Most of comparisons were held on TV material and MPEG2 codecs. Only recently appeared the comparisons which evaluate modern PC-oriented codecs that are able to operate on low bitrates.

    Goals of our assessment are subjective comparison of new versions of popular videocodecs, comparison of results with objective metrics and subjective assessment technology testing.

    Choice of video sequences and set of bitrates

    We were limited in amount of bitrates and video sequences, because each expert was to evaluate each sequence compressed with each bitrate at least once (for more details see "Method of the subjective assessment" below). Therefore we decided to concentrate on one application area of video codecs - films compression.

    Four scenes from "Terminator 2" and "The Matrix" were chosen: two of them with average motion and two with very fast one. Distortion on scenes with high level of motion is often the main factor of annoyance for a viewer of a compressed film. We chose two typical bitrates for film compression - 690 kbps and 1024 kbps(1).

    Encoding of sequences

    Sequences were encoded using one pass mode without subtle tuning of codecs' settings (most settings were left to default values). Only one parameter corresponding to the bitrate of compressed sequence was changed. Test sequences were about 10 seconds length, so the following technique was used to stabilize a codec: each sequence was repeated five times, compressed and the last repetition was cut from compressed sequence. This repetition was shown to the experts.

    Picture 1.    PSNR for repetitions of the "Battle" sequence, WMV 690 kbps

    Picture 2.    PSNR for repetitions of the "Battle" sequence, XviD 690 kbps

    On the PSNR graphs for repetitions of the "Battle" sequence it is clearly seen that the codec stabilizes on second repetition, and the results of the first pass are too optimistic.

    We used the "Direct Stream Copy" function from VirtualDub to cut the last repetition from a compressed sequence. Fortunately, all codecs on all sequences (except "Battle") made a key frame in the beginning of the last repetition, so it was possible to extract it. On the "Battle", probably due to fast motion, all codecs placed a key frame 10 frames before the last repetition, but this span is not big enough to influence results.

    According to the following table, all codecs managed to reach the desired bitrate with satisfactory precision.



    690 kbps

    1024 kbps

    Battle

    Matrix sc.1

    Matrix sc.2

    Rancho

    Battle

    Matrix sc.1

    Matrix sc.2

    Rancho

    DivX

    692

    690

    687

    688

    1044

    1020

    1026

    1028

    XviD

    681

    685

    680

    687

    1013

    1034

    1025

    1017

    x264

    688

    603

    687

    697

    1021

    907

    954

    1031

    WMV

    692

    693

    667

    697

    1034

    1049

    1046

    1045

    Method of the subjective assessment

    Idea of subjective comparison is to demonstrate video, processed with different sequences, to a group of experts and to record their impressions of video quality. There are a lot of subjective video quality assessment methods, many are described in recommendations of the ITU [1]. Unfortunately, most of them are targeted to TV material, and are not very convenient for conducting test on PC.

    For our assessment we chose SAMVIQ (Subjective Assessment of Multimedia Video Quality, [2]) test methodology. It was developed by the EBU (European Broadcasting Union) specifically for multimedia codecs comparison, it is easier to use and more convenient for subjective assessments on PC than other comparison methodologies. Among other subjective testing methods, it is implemented in the "MSU Perceptual Video Quality tool" [6]. This tool was used for the current comparison.

    Assessment steps:

    1. Expert inputs his or her name (any unique sequence of symbols).

    Picture 3.    Name input dialog

    2. Test for a color blindness (standard Ishihara test charts).

    Picture 4.    Color blindness test example

    3. For each version of test sequence:

    1) Reference (source) video is demonstrated.

    Picture 5.    Player view

    2) While there are still unwatched compressed versions of the current sequence, expert chooses a video, watches and rates it. Mark belongs to the segment from 0 to 100, the higher the better. Mark of a video that was already evaluated can be changed at any moment.

    Picture 6.    Rating dialog

    3) If all possible variants of test sequence are ranked, expert can finish it and proceed to the next test sequence. Expert can review any variant of a current test sequence any number of times.

    Different versions of test sequence are hidden beneath letters from A to I, so expert is not aware of codec that he is evaluating.

    Uncompressed (reference) video(2) is explicitly accessible through "Reference sequence" button, but is not evaluated when played with this button. In addition it is hidden among other letters and is evaluated along with compressed ones. Expert was ought to evaluate 9 versions of each sequence (4 codecs x 2 bitrates + 1 hidden reference video).

    Organization of the assessment

    The subjective assessment was carried out in two days. 50 experts took part in it, 14 of them were a video specialists. Three types of monitor were used, 6 x 15" CRT Dell, 1 x 17" CRT Samsung and 2 x 17" LCD Samsung, they were placed in two separated rooms (as stated in [8], monitor type has no significant influence on video quality testing). Up to 9 experts simultaneously took part in the assessment. Quiet atmosphere was maintained throughout the testing, monitors' settings were calibrated. All experts were instructed on the goals and the process of the testing.


    Sequences used in the assessment

    Battle

    Battle, frame 215

    Name

    Battle

    Resolution

    704x288

    Features

    Fragment of the "Terminator 2" movie. Difficult for compression: variable brightness, quick motion and frequent scene changes.


    Rancho

    Rancho, frame 149

    Name

    Rancho

    Resolution

    704x288

    Features

    Fragment from the "Terminator 2" movie. Slow motion in the beginning, then scene change followed by fast motion.


    Matrix sc.1

    Matrix sc.1, frame 178

    Name

    Matrix sc.1

    Resolution

    720x416

    Features

    Fragment from "The Matrix" movie. Consists of two parts: in the first one camera follows intruding soldiers, in the second one it is rotating. Motion is not very fast but is quite complicated.


    Matrix sc.2

    Matrix sc.2, frame 52

    Name

    Matrix sc.2

    Resolution

    720x416

    Features

    Fragment from "The Matrix" movie. Frame is filled with small parts of walls that are moving absolutely chaotically. Motion is not fast, but hardly predictable.


    Codecs used in the testing

    DivX 6.0

    MPEG4-ASP codec. "Create Bundle" package was used for the compression and the objective measurements, "Play Bundle" package was used for the subjective assessment.

    For encoding this codec was installed on a clean system, no codec settings except "Bitrate"-"kbit/s" were changed.

    XviD 1.1.0-beta2

    Open source MPEG4-ASP codec. We used the most recent version on the moment of the testing. Despite the fact that it is still in "beta" status, we did not experience any problems with it during the testing, and decided to use it because it was 1 dB better than the previous stable version. "Target bitrate" parameter was changed.

    Unfortunately, in the decoder of this version deblocking was disabled by default. It was not enabled due to the policy of non-intervention into codecs' parameters - this is the actual quality that an average viewer of an encoded film will get.

    x264 svn-352

    Open source H.264 codec. Was used only for compression. "Average Bitrate" parameter was changed.


    WMV 9.0

    Freeware codec of proprietary format. Decoder is included with latest versions of Windows. "Bit rate" parameter was changed

    ffdshow

    Was used only for decoding of stream compressed with x264, version of ff_x264.dll is 33, build data Aug 10 2005 16:33:17


    (1) 690 kbps - one movie in a medium quality for a CD, 1024 kbps - one movie in a high quality for two CDs or two movies for a DVD

    (2) Reference video was compressed with lossless codec Huffyuv v2.1.1

    Contents

  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • MOS+PSNR/bitrate graphs
  • Z-test resuts
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

  • Download



    Contact Information



    Additional information


    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:


    Other materials


    Video resources:

    Last updated: 12-May-2022


    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab