Perceptually Weighted Mean Squared Error Based Rate-Distortion Optimization for HEVC

Xiuzhe WU, Hanli WANG, Sudeng HU, Sam KWONG, C.-C. Jay KUO

Research output: Journal PublicationsJournal Article (refereed)peer-review

8 Citations (Scopus)


The process of rate-distortion (RD) optimization plays a key role for video coding, which aims to achieve a tradeoff between compression efficiency and video quality distortion. Although the conventional objective distortion metric mean squared error performs well in computational complexity, it is not always in accordance with the perceptual quality perceived by human visual system (HVS). Taking the characteristics of HVS into consideration, a perceptually weighted mean squared error (PWMSE) based RD model is proposed in this work. First, a low-pass filter is employed to process the original distortion information in order to simulate visual signal processing and obtain the perceptual distortion. Then, masking modulation in both temporal and spatial domains is introduced into distortion model, and a novel Lagrange multiplier is derived accordingly. The proposed PWMSE based RD model is applied to the high efficiency video coding standard, and comparative experimental results demonstrate its effectiveness. The project page can be found in
Original languageEnglish
Pages (from-to)824-834
JournalIEEE Transactions on Broadcasting
Issue number4
Early online date20 Oct 2020
Publication statusPublished - Dec 2020
Externally publishedYes

Bibliographical note

This work was supported in part by National Natural Science Foundation of China under Grant 61976159; in part by Shanghai Innovation Action Project of Science and Technology under Grant 20511100700; and in part by Shanghai Engineering Research Center of Industrial Vision Perception and Intelligent Computing under Grant 17DZ2251600. (Xiuzhe Wu and Hanli Wang are co-first authors.)


  • high efficiency video coding
  • Perceptual video coding
  • perceptually weighted mean squared error
  • rate-distortion optimization


Dive into the research topics of 'Perceptually Weighted Mean Squared Error Based Rate-Distortion Optimization for HEVC'. Together they form a unique fingerprint.

Cite this