Authors want to thank Stass Soldatov and Alexey Moskvin for help in preparation and verification of the report, and all of the experts who took part in a quality assessment for their priceless help in creation of this comparison.

In this comparison we evaluate the quality of four popular codecs on two bitrates (690 and 1024 kbps) using subjective assessment. We use SAMVIQ as a subjective testing methodology. An average subjective opinion on the quality of an encoded video is the most important characteristic of codec's performance. Fifty experts took part in the subjective assessment; their marks form resulting codec's rating. In addition, we measured objective metrics (PSNR, VQM and SSIM) and evaluated their ability to predict subjective opinion on our test set. Results of subjective evaluation prove that x264 codec of H.264 standard provides a significantly better subjective quality than other widely-spread codecs that took part in the comparison. It is shown that PSNR cannot always be reliable measure of video quality in practical cases (i.e., not specially created to corrupt the measure).

Codecs

Encoders' settings

Other settings were left to defaults. Default values are the values that are set after a codec is installed, you can see them on codecs' screenshots.

Decoders' settings

Decoders that are provided with codec were used for decoding of sequences of all codecs except for x264. For decoding files compressed with x264 we used popular tool "ffdshow", version of ff_x264.dll is 33 by Milan Cutca. All decoders' settings were let to defaults.

Sequences

The following table contains properties of the encoded sequences that were shown to experts (see "Encoding of sequences").

Goals of the subjective codecs assessment

During last few years many comparisons of video, audio and image codecs were carried out by our Graphics & Media Lab at Moscow State University (available at www.compression.ru/video). All of them used objective metrics like PSNR, VQM or SSIM. This fact has raised reasonable questions on adequacy of objective measures to the subjectively perceived quality.

Some organizations like VQEG (Video Quality Experts Group) and ITU (International Telecommunication Union) have held subjective assessments of video quality and evaluated adequacy of objective video quality metrics [5]. Most of comparisons were held on TV material and MPEG2 codecs. Only recently appeared the comparisons which evaluate modern PC-oriented codecs that are able to operate on low bitrates.

Goals of our assessment are subjective comparison of new versions of popular videocodecs, comparison of results with objective metrics and subjective assessment technology testing.

Choice of video sequences and set of bitrates

We were limited in amount of bitrates and video sequences, because each expert was to evaluate each sequence compressed with each bitrate at least once (for more details see "Method of the subjective assessment" below). Therefore we decided to concentrate on one application area of video codecs - films compression.

Four scenes from "Terminator 2" and "The Matrix" were chosen: two of them with average motion and two with very fast one. Distortion on scenes with high level of motion is often the main factor of annoyance for a viewer of a compressed film. We chose two typical bitrates for film compression - 690 kbps and 1024 kbps(1).

Encoding of sequences

Sequences were encoded using one pass mode without subtle tuning of codecs' settings (most settings were left to default values). Only one parameter corresponding to the bitrate of compressed sequence was changed. Test sequences were about 10 seconds length, so the following technique was used to stabilize a codec: each sequence was repeated five times, compressed and the last repetition was cut from compressed sequence. This repetition was shown to the experts.

Picture 1.    PSNR for repetitions of the "Battle" sequence, WMV 690 kbps

Picture 2.    PSNR for repetitions of the "Battle" sequence, XviD 690 kbps

On the PSNR graphs for repetitions of the "Battle" sequence it is clearly seen that the codec stabilizes on second repetition, and the results of the first pass are too optimistic.

We used the "Direct Stream Copy" function from VirtualDub to cut the last repetition from a compressed sequence. Fortunately, all codecs on all sequences (except "Battle") made a key frame in the beginning of the last repetition, so it was possible to extract it. On the "Battle", probably due to fast motion, all codecs placed a key frame 10 frames before the last repetition, but this span is not big enough to influence results.

According to the following table, all codecs managed to reach the desired bitrate with satisfactory precision.

Method of the subjective assessment

Idea of subjective comparison is to demonstrate video, processed with different sequences, to a group of experts and to record their impressions of video quality. There are a lot of subjective video quality assessment methods, many are described in recommendations of the ITU [1]. Unfortunately, most of them are targeted to TV material, and are not very convenient for conducting test on PC.

For our assessment we chose SAMVIQ (Subjective Assessment of Multimedia Video Quality, [2]) test methodology. It was developed by the EBU (European Broadcasting Union) specifically for multimedia codecs comparison, it is easier to use and more convenient for subjective assessments on PC than other comparison methodologies. Among other subjective testing methods, it is implemented in the "MSU Perceptual Video Quality tool" [6]. This tool was used for the current comparison.

Assessment steps:

1. Expert inputs his or her name (any unique sequence of symbols).

Picture 3.    Name input dialog

2. Test for a color blindness (standard Ishihara test charts).

Picture 4.    Color blindness test example

3. For each version of test sequence:

1) Reference (source) video is demonstrated.

Picture 5.    Player view

2) While there are still unwatched compressed versions of the current sequence, expert chooses a video, watches and rates it. Mark belongs to the segment from 0 to 100, the higher the better. Mark of a video that was already evaluated can be changed at any moment.

Picture 6.    Rating dialog

3) If all possible variants of test sequence are ranked, expert can finish it and proceed to the next test sequence. Expert can review any variant of a current test sequence any number of times.

Different versions of test sequence are hidden beneath letters from A to I, so expert is not aware of codec that he is evaluating.

Uncompressed (reference) video(2) is explicitly accessible through "Reference sequence" button, but is not evaluated when played with this button. In addition it is hidden among other letters and is evaluated along with compressed ones. Expert was ought to evaluate 9 versions of each sequence (4 codecs x 2 bitrates + 1 hidden reference video).

Organization of the assessment

The subjective assessment was carried out in two days. 50 experts took part in it, 14 of them were a video specialists. Three types of monitor were used, 6 x 15" CRT Dell, 1 x 17" CRT Samsung and 2 x 17" LCD Samsung, they were placed in two separated rooms (as stated in [8], monitor type has no significant influence on video quality testing). Up to 9 experts simultaneously took part in the assessment. Quiet atmosphere was maintained throughout the testing, monitors' settings were calibrated. All experts were instructed on the goals and the process of the testing.

Battle

Rancho

Matrix sc.1

Matrix sc.2

DivX 6.0

MPEG4-ASP codec. "Create Bundle" package was used for the compression and the objective measurements, "Play Bundle" package was used for the subjective assessment.

For encoding this codec was installed on a clean system, no codec settings except "Bitrate"-"kbit/s" were changed.

XviD 1.1.0-beta2

Open source MPEG4-ASP codec. We used the most recent version on the moment of the testing. Despite the fact that it is still in "beta" status, we did not experience any problems with it during the testing, and decided to use it because it was 1 dB better than the previous stable version. "Target bitrate" parameter was changed.

Unfortunately, in the decoder of this version deblocking was disabled by default. It was not enabled due to the policy of non-intervention into codecs' parameters - this is the actual quality that an average viewer of an encoded film will get.

x264 svn-352

Open source H.264 codec. Was used only for compression. "Average Bitrate" parameter was changed.

WMV 9.0

Freeware codec of proprietary format. Decoder is included with latest versions of Windows. "Bit rate" parameter was changed

ffdshow

Was used only for decoding of stream compressed with x264, version of ff_x264.dll is 33, build data Aug 10 2005 16:33:17