Hot news:

If you find a spelling error, please select an incorrect text and press Ctrl+Enter. Thank you!

Compression project >> Video Area Home

MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Return to Subjective Comparison of Modern Video Codecs home page!

Part 3. Evaluation of objective metrics

Contents

  • Introduction
  • Results of subjective comparison
  • Evaluation of objective metrics
  • Conclusions
  • Results of the assessment

    Correlation of objective metrics and subjective scores

    Below you can see two types of graphs: first one is a value of subjective score plotted against a value of an objective metric(4).

    Such graphs must be treated separately for each sequence, because subjective scores are context-sensitive (subjective score for a sequence is given by an expert according to another versions of it). These graphs are plotted together for the ease of perception.

    Picture 15.    Relation between PSNR and MOS

    As you can see, PSNR reflection of perceived video quality is limited. Sometimes one value of PSNR corresponds to absolutely different subjective ratings for the same sequence (marked with red oval) and vice versa (marked with grey oval).

    Picture 16.    Relation between VQM and MOS

    VQM produced prediction not better than PSNR on our test. Overall, quality is more or less predicted, but sometimes bad metric value corresponds to good perceived quality.

    Picture 17.    Relation between SSIM and MOS

    SSIM predicts subjective opinion with high precision, its' data is close to straight line for each sequence.

    Second type of graphs - subjective mark predicted with objective metric plotted against real subjective mark. Predicted mark was obtained by applying the fitting function for each sequence separately. Prediction is good when fitted values of objective metric are close to the straight line.

    Picture 18.    PSNR fitted to MOS for each sequence

    Picture 19.    VQM fitted to MOS for each sequence

    Picture 20.    SSIM fitted to MOS for each sequence

    As you can see, PSNR and VQM provided prediction of similar quality on our test set, and it was quite poor, meanwhile SSIM reached prediction close to the ideal one.

    To numerically evaluate prediction of the objective metrics, we calculated Pearson's correlation coefficient between objective marks (after applying the fitting function) and subjective ones. Correlation coefficient belongs to the segment from -1 to 1 and reflects degree of dependency between values (the higher the absolute value, the more powerful dependency is).

    Metric

    Correlation

    PSNR

    0.802

    VQM

    0.729

    SSIM

    0.937

    (4) PSNR, VQM and SSIM were measured with MSU Video Quality Measurement Tool[7]

    Contents

  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • MOS+PSNR/bitrate graphs
  • Z-test resuts
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

  • Download



    Contact Information



    Additional information


    Call for HEVC codecs 2019
    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:


    Other materials


    Video resources:

    Bookmark this page:   Add to Del.icio.us Add to Del.icio.us     Digg It Digg It     reddit reddit

     
    Last updated: 09-September-2019

    Search (Russian):
    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group


    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab