Joint Machine Learning and Game Theory for Rate Control in High Efficiency Video Coding

Wei GAO, Sam KWONG, Yuheng JIA

Research output: Journal PublicationsJournal Article (refereed)peer-review

56 Citations (Scopus)


In this paper, a joint machine learning and game theory modeling (MLGT) framework is proposed for inter frame coding tree unit (CTU) level bit allocation and rate control (RC) optimization in high efficiency video coding (HEVC). First, a support vector machine-based multi-classification scheme is proposed to improve the prediction accuracy of CTU-level rate-distortion (R-D) model. The legacy 'chicken-And-egg' dilemma in video coding is proposed to be overcome by the learning-based R-D model. Second, a mixed R-D model-based cooperative bargaining game theory is proposed for bit allocation optimization, where the convexity of the mixed R-D model-based utility function is proved, and Nash bargaining solution is achieved by the proposed iterative solution search method. The minimum utility is adjusted by the reference coding distortion and frame-level quantization parameter (QP) change. Finally, intra frame QP and inter frame adaptive bit ratios are adjusted to make inter frames have more bit resources to maintain smooth quality and bit consumption in the bargaining game optimization. Experimental results demonstrate that the proposed MLGT-based RC method can achieve much better R-D performances, quality smoothness, bit rate accuracy, buffer control results, and subjective visual quality than the other state-of-The-Art one-pass RC methods, and the achieved R-D performances are very close to the performance limits from the FixedQP method.
Original languageEnglish
Pages (from-to)6074-6089
JournalIEEE Transactions on Image Processing
Issue number12
Early online date25 Aug 2017
Publication statusPublished - Dec 2017
Externally publishedYes

Bibliographical note

This work was supported in part by the Natural Science Foundation of China under Grant 61672443, in part by the Hong Kong RGC General Research Fund 9042489 under Grant CityU 11206317, and in part by the Hong Kong RGC General Research Fund 9042322 under Grant CityU 11200116.


  • bit allocation
  • game theory
  • H.265/HEVC
  • iterative search
  • machine learning
  • Nash bargaining solution (NBS)
  • non-numerical solution
  • R-D model classification
  • rate control
  • support vector machine (SVM)
  • video coding


Dive into the research topics of 'Joint Machine Learning and Game Theory for Rate Control in High Efficiency Video Coding'. Together they form a unique fingerprint.

Cite this