MSU Subjective Comparison of Modern Video Codecs

MSU Graphics & Media Lab (Video Group)

Return to Subjective Comparison of Modern Video Codecs home page!

Part 3. Evaluation of objective metrics

Contents

  • Introduction
  • Results of subjective comparison
  • Evaluation of objective metrics
  • Conclusions

    • Results of the assessment

    Correlation of objective metrics and subjective scores

    Below you can see two types of graphs: first one is a value of subjective score plotted against a value of an objective metric(4).

    Such graphs must be treated separately for each sequence, because subjective scores are context-sensitive (subjective score for a sequence is given by an expert according to another versions of it). These graphs are plotted together for the ease of perception.

    Picture 15.    Relation between PSNR and MOS

    As you can see, PSNR reflection of perceived video quality is limited. Sometimes one value of PSNR corresponds to absolutely different subjective ratings for the same sequence (marked with red oval) and vice versa (marked with grey oval).

    Picture 16.    Relation between VQM and MOS

    VQM produced prediction not better than PSNR on our test. Overall, quality is more or less predicted, but sometimes bad metric value corresponds to good perceived quality.

    Picture 17.    Relation between SSIM and MOS

    SSIM predicts subjective opinion with high precision, its' data is close to straight line for each sequence.

    Second type of graphs - subjective mark predicted with objective metric plotted against real subjective mark. Predicted mark was obtained by applying the fitting function for each sequence separately. Prediction is good when fitted values of objective metric are close to the straight line.

    Picture 18.    PSNR fitted to MOS for each sequence

    Picture 19.    VQM fitted to MOS for each sequence

    Picture 20.    SSIM fitted to MOS for each sequence

    As you can see, PSNR and VQM provided prediction of similar quality on our test set, and it was quite poor, meanwhile SSIM reached prediction close to the ideal one.

    To numerically evaluate prediction of the objective metrics, we calculated Pearson's correlation coefficient between objective marks (after applying the fitting function) and subjective ones. Correlation coefficient belongs to the segment from -1 to 1 and reflects degree of dependency between values (the higher the absolute value, the more powerful dependency is).

    Metric

    Correlation

    PSNR

    0.802

    VQM

    0.729

    SSIM

    0.937

    (4) PSNR, VQM and SSIM were measured with MSU Video Quality Measurement Tool[7]

    Contents

  • Introduction
  • Gratitude
  • Comparison abstract
  • Overview
  • Rules and goals
  • Sequences
  • Codecs
  • Results of subjective comparison
  • MOS+PSNR/bitrate graphs
  • Z-test resuts
  • MOS+PSNR graphs, grouped by bitrate
  • Evaluation of objective metrics
  • Correlation of objective metrics and subjective scores
  • Conclusions
  • Average MOS for all codecs
  • General conclusions

    • Download

    Contact Information

    Additional information

    See all MSU Video Codecs Comparisons

    MSU video codecs comparisons resources:

    Other materials

    Video resources:

    3D and stereo video
    Projects on 3D and stereo video processing and analysis
    MSU Video Quality Measurement tools
    Programs with different objective and subjective video quality metrics implementation     		Codecs comparisons
    Objective and subjective quality evaluation
    tests for video and image codecs
    Ext. link: x264 parameters efficiency comparison
    Public MSU video filters
    Here are available VirtualDub and AviSynth filters. For a given type of digital video filtration we typically develop a family of different algorithms and implementations. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru.     		Filters for companies
    We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision.
    Video codecs projects
    Different research and development
    projects on video codecs     		Other
    Other information
    Last updated: 12-May-2022

    Server size: 8069 files, 1215Mb (Server statistics)

    Project updated by
    Server Team and MSU Video Group

    Project sponsored by YUVsoft Corp.

    Project supported by MSU Graphics & Media Lab