Generative Adversarial Network-Based Intra Prediction for Video Coding

Linwei ZHU, Sam KWONG, Yun ZHANG, Shiqi WANG, Xu WANG

Research output: Journal PublicationsJournal Article (refereed)peer-review

40 Citations (Scopus)

Abstract

In this paper, a novel intra prediction method is proposed to improve the video coding performance, in which the generative adversarial network (GAN) is adopted to intelligently remove the spatial redundancy with the inference process. The proposed GAN-based method improves the prediction by exploiting more information and generating more flexible prediction patterns. In particular, the intra prediction is modeled as an inpainting task, which is accomplished with the GAN model to fill in the missing part by conditioning on the available reconstructed pixels. As such, the learned GAN model is incorporated into both video encoder and decoder, and the rate-distortion optimization is performed for the competition between GAN-based intra prediction and traditional angular-based intra prediction to achieve better coding performance. The proposed scheme is implemented into the high-efficiency video coding test model (HM 16.17) and the versatile video coding test model (VTM 1.1). The experimental results show that the proposed algorithm can achieve 6.6%, 7.5%, and 7.5% under HM 16.17 and 6.75%, 7.63%, and 7.65% under VTM 1.1 bit rate savings on average for luma and chroma components in the intra coding scenario.
Original languageEnglish
Pages (from-to)45-58
JournalIEEE Transactions on Multimedia
Volume22
Issue number1
Early online date24 Jun 2019
DOIs
Publication statusPublished - Jan 2020
Externally publishedYes

Bibliographical note

This work was supported in part by the Natural Science Foundation of China under Grant 61672443, Grant 61871312, and Grant 61871270, in part by China Postdoctoral Science Foundation under Grant 2019M653127, in part by Hong Kong RGC General Research Fund 9042322 (CityU 11200116) and 9042489 (CityU 11206317), in part by Hong Kong RGC Early Career Scheme 9048122 (CityU 21211018), in part by City University of Hong Kong under Grant 7200539/CS, in part by Guangdong Natural Science Funds for Distinguished Young Scholar under Grant 2016A030306022, in part by Shenzhen Science and Technology Development Project under Grant JCYJ20170811160212033, in part by Shenzhen International Collaborative Research Project under Grant GJHZ20170314155404913, in part by Shenzhen Science and Technology Plan Project under Grant JCYJ20180507183823045, in part by Guangdong Provincial Science and Technology Development under Grant 2017B010110014, in part by Free Application Fund of Natural Science Foundation of Guangdong Province under Grant 2018A0303130126, in part by Guangdong International Science and Technology Cooperative Research Project under Grant 2018A050506063 and in part by Membership of Youth Innovation Promotion Association, Chinese Academy of Sciences under Grant 2018392.

Keywords

  • Generative adversarial network
  • high efficiency video coding
  • inpainting
  • intra prediction
  • versatile video coding

Fingerprint

Dive into the research topics of 'Generative Adversarial Network-Based Intra Prediction for Video Coding'. Together they form a unique fingerprint.

Cite this