MPEG-4 AVC/H.264 Video Codecs Comparison 2010 - Appendixes
MSU Graphics & Media Lab (Video Group)
Appendix 1. Subjective Comparison and Psycho-visual Enhancements
Brief Description
In this work, we analyzed video codecs not only with help of objective metrics, but also using subjective comparisons using SAMVIQ (Subjective Assessment Method for Video Quality evaluation) methodology. This new method was created by the EBU (European Broadcasting Union).
Another task in this part of comparison was to analyze the psycho-visual enhancement quality during encoding. We tested five video codecs, and one codec was tested with two almost identical presets: one without psycho-visual enhancement and another with psycho-visual enhancement.
SAMVIQ Description
During testing, each expert is able to play any sequence from the test set and give it a mark, and he is able to play a reference video. Marks are in the range of 0 to 100. More information about the methodology can be found in the publication SAMVIQ: A new EBU methodology for video quality evaluations in multimedia(Kozamernik, F., Steinmann, V., Sunna, P. and Wyckens, E., SMPTE journal, 2005 04 April).Subjective Assessment Description
Forty-two experts participated in this subjective assessment. The experts represent a variety of individuals: males and females of ages 18 to 40 who are PC users or video and graphics experts. Each user watched five video groups, with eight videos in each group. Each video group contains six encoded videos and one reference video (the viewers were unaware of which type they were viewing), as well as a reference video specified as such. Users gave each video a mark of 0 to 100, and the marks for all users and sequences were averaged. Some extreme results were discarded before obtaining the final result.Video Sequences
Five different video sequences were used in this comparison."Battle"
This sequence is a fragment from the beginning of the movie Terminator 2. The compression of this sequence is the most difficult among all of the sequences in the analysis. This difficulty is mainly due to three reasons: continual brightness variation (resulting from explosions and laser flashes as seen in the picture above), very fast motion and frequent scene changes. These characteristics often cause codecs to compress frames as I-frames."Foorball"
This sequence is a part of sports telecast. The sequence has a high contrast level, strong motion and rich colors."Italy"
This sequence has many scenes with fading transitions between them. This fading can cause encoding difficulties. The sequence contains many sharp details."Quadbike"
This sequence has slow motion with an almost static camera and few scene changes."Simpsons"
This sequence is a part of the Simpsons animated movie; it has high contrast and different types of motion.Video Codecs
Five codecs were used in the comparison:Results
Full results could be found in professional version of this report: H.264 Comparison Report Pro 2010 - Enterprise editionThis version contains only one-sequence results
"Football" Sequence
For the "Football" sequence, x264 provides the highest quality. x264 with psycho-visual enhancement provides very similar (almost undistinguishable) results. The codecs can be rated by visual quality as follows:- x264 with psycho-visual enhancement
- x264
- MainConcept
- DivX H.264
- Elecard
- DivX ASP
H.264 Comparison Report Pro 2010 - Enterprise edition
Appendix 2. x264 Over-years Comparison
The quality of an H.264 codec, over several years, can be compared for a given video sequence. The x264 encoder was chosen for this task because it is present in almost every MSU VIDEO MPEG-4 AVC/H.264 codec comparison, and it produces good results compared with other encoders. Next figures show the position of the x264 codec compared with other codecs for the “Battle” sequence. For all years except 2005, x264 shows the best results. For years 2006–2009, we have shown results using Y-SSIM as the quality metric; for 2005, we did not use this as the main metric. In light of these results, x264 could be a good reference encoder for analyzing the overall progress of H.264 encoders over time. Next figure shows the RD curve for the “Battle” sequence using x264 encoders from different years. The best encoder is this year’s x264; the worst is the 2005 version.
Using SSIM, the codecs can be ranked as follows:
- x264 (2010)
- x264 (2009)
- x264 (2007)
- x264 (2006)
- x264 (2005)
Interestingly, if Y-PSNR is used as the quality metric, x264 does not exhibit the kind of progress shown in previous figure. This difference is because for past years, the x264 developers specified, for use in testing, the presets optimized for the SSIM metric. Also, note that no encoder (among the different x264 versions) produces the best results simultaneously for both SSIM and PSNR.
The bitrate handling mechanism for the x264 encoder is quite good for each version, as following figure indicates. Results for previous x264 versions (0.98 of target bitrate) could be explained by a different interpretation of kbps (1,024 versus 1,000 bits per second).
The per-frame analysis presented in following figure shows that the main encoding mechanism did not changed significantly.
Appendix 3. Theora vs. x264 Single-thread Comparison
Multithreaded encoding is not supported by the Theora encoder, but CPUs with four cores are used for the tests. This situation leads to an incorrect speed comparison between Theora and other codecs. To eliminate this disparity, we have turned on the x264 encoder’s single-thread mode and compared the results with those of Theora. The overall speed/quality trade-off for “Movies” and “HDTV” use cases is depicted in next figures.
The main conclusion is that even in single-thread mode, Theora is a little bit slower and produces much poorer quality than does x264.
This sequence is a fragment from the Ice Age 3 animated movie. This movie has low-contrast portions and high-contrast portions, and it has many types of motion: camera panning, slow motion and very fast motion. Also, it has a scene with colors that differ completely from those of other scenes. Small black letterboxes appear at the top and bottom of the video.
This sequence is a fragment from the Indiana Jones movie. Compression of this sequence is difficult for two main reasons: the presence of low-contrast scenes and the high level of motion in different scenes. Also, several scenes have very different types of motion, ranging from almost static scenes with talking people to scenes with strong motion (for example, the scene where stones fall).
This sequence is a fragment from the Enemy of the State movie. This sequence includes outdoor scenes with strong motion at the beginning when the bicyclist runs, as well as scenes with low motion and indoor scenes with normal motion. This sequence has scenes with different lighting conditions.
This sequence is a fragment from the Up animated movie. The sequence contains low-contrast scenes with almost static brightness and high-colored scenes, and it contains many scenes with a few frames that include quick scene changes.
This sequence contains scenes with the camera panning over a landscape view of the Amazon; also, some scene changes take place with the camera panning again. The video frames consist of many edges and high-contrast, sharp details, such as leaves and branches. This sequence was downloaded from the Microsoft website:
http://www.microsoft.com/windows/windowsmedia/musicandvideo/hdvideo/contentshowcase.aspx
This sequence is a part of the Iron Man 2 movie trailer. It has low-brightness scenes at the beginning, followed by a scene with very fast motion and scenes with slow camera panning. Some frames are very blurry. Also, the sequence contains scenes with flashing lights, which could be a big problem for some encoders. Black letterboxes appear at the top and bottom of the video.
This close-up sequence is similar to “Mobile&Calendar” and includes a moving calendar with text and a detailed photo of the Vasa ship. It also includes a moving train with colorful toys. The background has two types of wallpaper: one is brown with details, and the other is yellow with drawn figures. The sequence is very detailed and is normally demanding. The main potential compression difficulty is the many small, sharp details on the calendar and on the background.
This sequence is a fragment of the “Troy” movie and contains three parts with sharp scene changes. The video includes medium scene motion and slow camera motion. In terms of compression, this sequence is difficult to compress because of the many small details.
MSU video codecs comparisons resources:
Appendix 4. Test Set of Video Sequences
Movie Sequences
"Ice Age"
"Indiana Jones"
"State Enemy"
"Up"
HDTV Sequences
"Amazon"
"Iron Man"
"Mobile Calendar"
"Troy"
Appendix 5. Tested Codecs and Presets
Codecs
DivX AVC/H.264 Video Encoder
Elecard AVC Video Encoder 8-bit edition
Intel® MediaSDK AVC/H.264
MainConcept AVC/H.264 Video Encoder Console Application
Microsoft Expression Encoder 3
Theora encoder
x264
Presets
The table below lists the settings used in this comparison for all of the codecs.
Contact Information
Other Materials
Video resources:
Server size: 8069 files, 1215Mb (Server statistics)
Project updated by
Server Team and
MSU Video Group
Project sponsored by YUVsoft Corp.
Project supported by MSU Graphics & Media Lab