MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Return to Subjective Comparison of Modern Video Codecs home page!

Part 4. Conclusions

Contents

  • Introduction
  • Results of subjective comparison
  • Evaluation of objective metrics
  • Conclusions
  • Results of the comparison

    Following graphs present MOS value for each codec after averaging among all sequences and all bitrates.

    Picture 21.    Average MOS for all codecs

    Picture 22.    Average MOS for all codecs and bitrates


    Plots from previous part of report reflect average subjective opinion on codecs. To calculate rank of each codec on the whole test set, we apply following method: count rank of codec for each sequence and then count sum of ranks ("Overall" row). Sorting this array we get rank of codec in comparison (Ref. - uncompressed sequence, was evaluated by experts as well as others).

    Battle

    Rancho

    Matrix sc.1

    Matrix sc.2

    Overall

    Place

    Ref.

    1

    1

    2

    1

    5

    -

    x264 1024

    2

    3

    1

    2

    8

    1

    DivX 1024

    3

    4

    5

    3

    15

    2

    XviD 1024

    7

    2

    3

    5

    17

    3

    x264 690

    5

    6

    4

    4

    19

    4

    WMV 1024

    4

    5

    6

    6

    21

    5

    DivX 690

    6

    8

    7

    7

    28

    6

    WMV 690

    8

    7

    9

    9

    33

    7

    XviD 690

    9

    9

    8

    8

    34

    8

    x264 supremacy is obvious for both 690 kbps and 1024 kbps, it is interesting that its' average mark for 690 kbps is better than the mark of WMV on 1024 kbps. DivX is the second, and WMV and XviD are last. Last place of XviD can be explained by the lack of deblocking during the assessment.

    Conclusions

    • Open source x264 codec of the new H.264 compression standard was found out to be better than long-developed proprietary solutions.
    • Subjective comparison is stable and productive method for video systems' assessment when all testing conditions are precisely adhered to.
    • Subjective tests can not always be replaced by objective ones with sufficient precision. SSIM was the best objective metric in our comparison.

    References

    1. International Telecommunication Union, "Methodology for the subjective assessment of the quality of television pictures (ITU-R BT.500-11)" (2002)

    2. European Broadcasting Union, "SAMVIQ - a New EBU Methodology for Video Quality Evaluations In Multimedia", web site www.ebu.ch

    3. Feng Xiao, "DCT-based Video Quality Evaluation", Final Project for EE392J (2000)

    4. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity", IEEE Transactions on Image Processing, vol. 13, no. 4 (2004)

    5. VQEG, "Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment" (2000), web site www.vqeg.org

    6. MSU Graphics & Media Lab (Video Group), "MSU Perceptual Video Quality Tool" www.compression.ru/video/quality_measure/perceptual_video_quality_tool_en.html

    7. MSU Graphics & Media Lab (Video Group), "MSU Video Quality Measurement Tool" www.compression.ru/video/quality_measure/video_measurement_tool_en.html

    8. M.H. Prinson, S. Wolf, "The Impact of Monitor Resolution and Type on Subjective Video Quality Testing", NTIA TM-04-412

    9. Avery Lee, "VirtualDub", web site www.virtualdub.org


    Contents

  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • MOS+PSNR/bitrate graphs
  • Z-test resuts
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

  • Download



    Contact Information



    Additional information


    Call for cloud encoding comparison participation 2020
    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:


    Other materials


    Video resources:

    Last updated: 30-April-2020


    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab