MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Return to Subjective Comparison of Modern Video Codecs home page!

Part 3. Evaluation of objective metrics


  • Introduction
  • Results of subjective comparison
  • Evaluation of objective metrics
  • Conclusions
  • Results of the assessment

    Correlation of objective metrics and subjective scores

    Below you can see two types of graphs: first one is a value of subjective score plotted against a value of an objective metric(4).

    Such graphs must be treated separately for each sequence, because subjective scores are context-sensitive (subjective score for a sequence is given by an expert according to another versions of it). These graphs are plotted together for the ease of perception.

    Picture 15.    Relation between PSNR and MOS

    As you can see, PSNR reflection of perceived video quality is limited. Sometimes one value of PSNR corresponds to absolutely different subjective ratings for the same sequence (marked with red oval) and vice versa (marked with grey oval).

    Picture 16.    Relation between VQM and MOS

    VQM produced prediction not better than PSNR on our test. Overall, quality is more or less predicted, but sometimes bad metric value corresponds to good perceived quality.

    Picture 17.    Relation between SSIM and MOS

    SSIM predicts subjective opinion with high precision, its' data is close to straight line for each sequence.

    Second type of graphs - subjective mark predicted with objective metric plotted against real subjective mark. Predicted mark was obtained by applying the fitting function for each sequence separately. Prediction is good when fitted values of objective metric are close to the straight line.

    Picture 18.    PSNR fitted to MOS for each sequence

    Picture 19.    VQM fitted to MOS for each sequence

    Picture 20.    SSIM fitted to MOS for each sequence

    As you can see, PSNR and VQM provided prediction of similar quality on our test set, and it was quite poor, meanwhile SSIM reached prediction close to the ideal one.

    To numerically evaluate prediction of the objective metrics, we calculated Pearson's correlation coefficient between objective marks (after applying the fitting function) and subjective ones. Correlation coefficient belongs to the segment from -1 to 1 and reflects degree of dependency between values (the higher the absolute value, the more powerful dependency is).









    (4) PSNR, VQM and SSIM were measured with MSU Video Quality Measurement Tool[7]


  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • MOS+PSNR/bitrate graphs
  • Z-test resuts
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

  • Download

    Contact Information

    Additional information

    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:

    Other materials

    Video resources:

    Last updated: 12-May-2022

    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab