Call for third MSU HEVC codecs comparison - 2017

Twelfth modern video codec comparison
For real researchers, developers and professional users in field of high-end video compression

MSU Graphics & Media Lab (Video Group)

Important Dates

March, 31 Deadline for receipt of a codec with required presets
April, 25 Deadline for settling technical problems with codec’s functioning
August, 7 Draft version of report that will be sent to all participants
August, 14 Deadline for reception of comments to the draft
August, 25 Comparison report release

Task of the Comparison


To perform comparative unbiased analysis of the current software and hardware (GPU-based) implementations of HEVC/H.265 video coding standard and compare it to the best implementations of other video coding standards (H.264/MPEG-4 AVC, VP9, VP8 and MPEG-4) using objective metrics.

Scope of Test

Summary report topics:
  • Objective measurements
  • Encoding time
  • Bitrate keeping
  • Speed/Quality trade-off analysis
  • Averaged objective results analysis
  • Leaders in different use-cases
Comparison methodology main points:
  • 2 metrics: SSIM and PSNR
  • 3 color-planes (Y,U,V) and integral metric values
  • 25-30 HD video sequences (main report) + 10-12 4K video sequences (report appendix)
  • prosumer-level modern hardware
  • 8-10 different target bitrates (1-12 Mbps for HD and 2-16 Mbps for 4K)
  • 3 various use-cases (Fast, Universal and Ripping) differ by speed/quality trade-off
  • fully automatic testing system
  • 3000-4000 result figures

Software and methodology for encoder analysis

MSU team hasup to 20 years of experience in video codec analysis, testing, optimization. Here are some facts about the Previous MSU Video Codecs Comparisons:
  • There were more than 370.000 downloads of previous H.264 and HEVC video codec comparisons results
  • Many codec's bugs were found and reported to developers
  • More than 20 private reports for codec developers (describe weak and strong points for codec) after public report versions
  • Here you can see Selected comments for MPEG-4 AVC/H.264 video codecs comparison
    •  In addition, you can check out some useful links about previous video codec comparisons:
  • MSU HEVC/H.265 Video Codec Comparison - 2016
  • Eighth Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
  • See all our comparisons
    •

    New in this Comparison

  • New video content - we change video sequence test set every year. This year we will use content classification system presented last year HEVC comparison.
  • Subjective quality assessments: we plan to make additional subjective codec comparison.
  • GPU codecs could be analyzed:Nvidia, AMD, and Skylake based encoders could be analyzed. We would analyze GPU-encoders in case of encoder developers interest. You can send us your GPU-accelerated or GPU-based encoder with or without CPU-based encoder!

    • Comparison Rules

    This year encoder developers send us a bundle of same presets (with different speed/quality characteristics) for all use-cases. Please pay attention that we will use multi-core CPU for encoding, so you can use multi-threading.
  • Decoding is performed with reference decoder.
  • We do not limit GOP size and intra-period.
  • Before results' publishing each developer will receive the results of its codec and competitive free codecs. Developers of each codec can write a comment (one paragraph) about the comparison results. That comment will be included in the report.
  • We are willing to completely or partially delete information about some codec in the public version of comparison report only in exceptional cases (e.g. critical errors in a codec).
  • The participation is free with results publishing
  • You can join comparison for free if you agree that your codec's results will be published.
  • If your company wants to receive results of your codec testing with possibility to exclude it results from publication and information disclosure, you should pay for measurements and report preparing before comparison begins.
  • Enterprise version of comparison report is available for direct participants for free
  • All participants will receive following deliverables to verify the results for free:
  • video sequences used in comparison
  • binaries of all free encoders used in comparison to verify the results
  • all raw video quality metric and encoding speed data for its encoder and for all of free encoders used in comparison

    • Test Hardware Characteristics
    Next hardware for codec testing will be used (the same as in HEVC Video Codecs Comparison - 2016):
    • CPU: Intel Socket 1151 Core i7 6700K (Skylake) (4Ghz, 4C8T, 1Mb+8Mb L3, 8 GT/s, HD Graphics 530, TDP 91W)
    • Mainboard: ASUS Z170-K ATX
    • RAM: Kingston KVR21N15D8/8 8Gb DIMM DDR4 PC17000/2133MHz CL15 1.2v Dual Rank x8
    • SSD: Intel 535 series 480Gb SSD SATA2.5"
    • HDD: Western Digital WD4003FZEX 4Tb (SATA 6Gb/s, 7200rpm, 64Mb cache)
    • SVGA: ASUS STRIX-GTX960-DC2-4GD5 4Gb PCI-Ex16 3.0 128-bit GDDR5 DVI+HDMI+3xDP

    Codec Requirements

  • Presets for different types of video sequences should be provided by the developers
  • Codec should allow to set arbitrary bitrate of resulting stream
  • Preferable codec interface - console codec version (with batch processing support — bitrate and file names must be possible to assign from the command line).
  • Encoder should be compatible with reference decoder

    • Encoding speed requirements
    For encoder alignment selected presets should provide following encoding speed:
    All speed requirements are for 1080p sequence encoded at 5Mbps: