Abstract
The quality of synthesized view plays an important role in the three dimensional (3D) video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN) based synthesized view quality enhancement method for 3D High Efficiency Video Coding (HEVC) is proposed. Firstly, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Secondly, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Thirdly, a new Lagrange multiplier in the rate-distortion (RD) cost function is derived to adapt the CNN based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) index, which significantly outperforms the state-of-theart methods.
| Original language | English |
|---|---|
| Pages (from-to) | 5365-5377 |
| Journal | IEEE Transactions on Image Processing |
| Volume | 27 |
| Issue number | 11 |
| Early online date | 20 Jul 2018 |
| DOIs | |
| Publication status | Published - Nov 2018 |
| Externally published | Yes |
Bibliographical note
This work was supported in part by the Natural Science Foundation of China under Grant 61672443, Grant 61471348, and Grant 61571274, in part by the Hong Kong RGC General Research Fund under Grant 9042322 (CityU 11200116) and Grant 9042489 (CityU 11206317), in part by the Hong Kong RGC Early Career Scheme under Grant 9048122 (CityU 21211018), in part by the City University of Hong Kong under Grant 7200539/CS, in part by the Guangdong Natural Science Funds for Distinguished Young Scholar under Grant 2016A030306022, in part by the Project for Shenzhen Science and Technology Development under Grant JSGG20160229202345378, in part by the Shenzhen International Collaborative Research Project under Grant GJHZ20170314155404913, in part by the Guangdong Provincial Science and Technology Development under Grant 2017B010110014, in part by the Guangdong Special Support Program for Youth Science and Technology Innovation Talents under Grant 2014TQ01X345, in part by the Membership of Youth Innovation Promotion Association, Chinese Academy of Sciences, under Grant 2018392, and in part by the Shandong Natural Science Funds for Distinguished Young Scholar under Grant JQ201614.Keywords
- 3D high efficiency video coding
- Convolutional neural network
- depth coding
- Lagrange multiplier
- view synthesis