MSU Subjective Comparison of Modern Video CodecsMSU Graphics & Media Lab (Video Group)
Part 3. Evaluation of objective metrics
Results of the assessment
Correlation of objective metrics and subjective scores
Below you can see two types of graphs: first one is a value of subjective score plotted against a value of an objective metric(4).
Such graphs must be treated separately for each sequence, because subjective scores are context-sensitive (subjective score for a sequence is given by an expert according to another versions of it). These graphs are plotted together for the ease of perception.
Picture 15. Relation between PSNR and MOS
As you can see, PSNR reflection of perceived video quality is limited. Sometimes one value of PSNR corresponds to absolutely different subjective ratings for the same sequence (marked with red oval) and vice versa (marked with grey oval).
Picture 16. Relation between VQM and MOS
VQM produced prediction not better than PSNR on our test. Overall, quality is more or less predicted, but sometimes bad metric value corresponds to good perceived quality.
Picture 17. Relation between SSIM and MOS
SSIM predicts subjective opinion with high precision, its' data is close to straight line for each sequence.
Second type of graphs - subjective mark predicted with objective metric plotted against real subjective mark. Predicted mark was obtained by applying the fitting function for each sequence separately. Prediction is good when fitted values of objective metric are close to the straight line.
Picture 18. PSNR fitted to MOS for each sequence
Picture 19. VQM fitted to MOS for each sequence
Picture 20. SSIM fitted to MOS for each sequence
As you can see, PSNR and VQM provided prediction of similar quality on our test set, and it was quite poor, meanwhile SSIM reached prediction close to the ideal one.
To numerically evaluate prediction of the objective metrics, we calculated Pearson's correlation coefficient between objective marks (after applying the fitting function) and subjective ones. Correlation coefficient belongs to the segment from -1 to 1 and reflects degree of dependency between values (the higher the absolute value, the more powerful dependency is).
(4) PSNR, VQM and SSIM were measured with MSU Video Quality Measurement Tool
- MSU Subjective Comparison of Contemporary Video Codecs - PDF (853 kB)
- MSU Subjective Comparison of Contemporary Video Codecs - ZIP (721 kB)
See all MSU Video Codecs Comparisons
MSU video codecs comparisons resources:
- Introduction to Video Codecs Comparison
- Lossless Video Codecs Comparison 2004 (October 2004)
- MPEG-4 SP/ASP Video Codecs Comparison (March 2005)
- JPEG 2000 Image Codecs Comparison (September 2005)
- First Annual MPEG-4 AVC/ H.264 Video Codecs Comparison (January 2005)
- Second Annual MPEG-4 AVC/H.264 Video Codec Comparison (December 2005)
- Subjective Comparison of Modern Video Codecs (February 2006)
- MPEG-2 Video Decoders Comparison (May 2006)
- WMP and JPEG2000 Comparison (October 2006)
- Third Annual MPEG-4 AVC/H.264 Comparison (December 2006) (All versions for free!)
- Lossless Video Codecs Comparison 2007 (March 2007)
- Fourth Annual MPEG-4 AVC/H.264 Comparison (December 2007) (All versions for free!)
- Options Analysis of MPEG-4 AVC/H.264 Codec x264 (December 2008)
- Fifth MPEG-4 AVC/H.264 Comparison (May 2009) (All versions for free!)
- Sixth MPEG-4 AVC/H.264 Comparison (May 2010)
- Seventh MPEG-4 AVC/H.264 Comparison (May 2011)
- Eighth MPEG-4 AVC/H.264 Comparison (May 2012)
- Ninth MPEG-4 AVC/H.264 Comparison (Dec 2013)
- Tenth Video Codec Comparison (HEVC) (Oct 2015)
- Eleventh Video Codec Comparison (HEVC) (Aug 2016)
- Twelfth Video Codec Comparison (HEVC) (Aug 2017)
- Thirteen Video Codec Comparison (HEVC) (Aug 2018)
- Fourteen Video Codec Comparison (HEVC) (Sept 2019)
- Cloud Encoding Servoces Comparison 2019 (Dec 2019)
- Fifteen Video Codec Comparison (HEVC) (Dec 2020)
- Codec Analysis for Companies:
Server size: 8069 files, 1215Mb (Server statistics)
Project sponsored by YUVsoft Corp.
Project supported by MSU Graphics & Media Lab