Reinforcement learning based coding unit early termination algorithm for high efficiency video coding

Na LI, Yun ZHANG, Linwei ZHU, Wenhan LUO, Sam KWONG

Research output: Journal PublicationsJournal Article (refereed)peer-review

19 Citations (Scopus)


In this paper, we propose a Reinforcement Learning (RL) based Coding Unit (CU) early termination algorithm for High Efficiency Video Coding (HEVC). RL is utilized to learn a CU early termination classifier independent of depths for low complexity video coding. Firstly, we model the process of CU decision as a Markov Decision Process (MDP) according to the Markov property of CU decision. Secondly, based on the MDP, a CU early termination classifier independent of depths is learned from trajectories of CU decision across different depths with the end-to-end actor-critic RL algorithm. Finally, a CU decision early termination algorithm is introduced with the learned classifier, so as to reduce computational complexity of CU decision. We implement the proposed scheme with different neural network structures. Two different neural network structures are utilized in the implementation of RL based video encoder, which are evaluated to reduce video coding complexity by 34.34% and 43.33%. With regard to Bjøntegaard delta peak signal-to-noise ratio and Bjøntegaard delta bit rate, the results are −0.033 dB and 0.85%, −0.099 dB and 2.56% respectively on average under low delay B main configuration, when compared with the HEVC test model version 16.5.
Original languageEnglish
Pages (from-to)276-286
JournalJournal of Visual Communication and Image Representation
Early online date20 Feb 2019
Publication statusPublished - Apr 2019
Externally publishedYes

Bibliographical note

This work was supported in part by the National Natural Science Foundation of China under Grant 61471348, 61871372, 61672443, in part by Guangdong Natural Science Foundation for Distinguished Young Scholar under Grant 2016A030306022, in part by Shenzhen Science and Technology Development Project under Grant JCYJ20170811160212033 and Shenzhen International Collaborative Research Project under Grant GJHZ20170314155404913, in part by the Key Project for Guangdong Provincial Science and Technology Development under Grant 2017B010110014, in part by Free Application Fund of Natural Science Foundation of Guangdong Province under Grant 2018A0303130126, in part by RGC General Research Fund (GRF) 9042322, 9042489 ( CityU 11200116, 11206317 ), in part by Guangdong International Science and Technology Cooperative Research Project under Grant 2018A050506063, in part by Membership of Youth Innovation Promotion Association, Chinese Academy of Sciences under Grant 2018392 .


  • Actor-critic
  • Coding tree unit
  • Early termination
  • High efficiency video coding
  • Markov decision processing
  • Reinforcement learning


Dive into the research topics of 'Reinforcement learning based coding unit early termination algorithm for high efficiency video coding'. Together they form a unique fingerprint.

Cite this