Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion

Jing JIN, Junhui HOU, Jie CHEN, Huanqiang ZENG, Sam KWONG, Jingyi YU

Research output: Journal PublicationsJournal Article (refereed)peer-review

64 Citations (Scopus)

Abstract

A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement.
Original languageEnglish
Pages (from-to)1819-1836
Number of pages18
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume44
Issue number4
Early online date23 Sept 2020
DOIs
Publication statusPublished - Apr 2022
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.

Funding

This work was supported in part by the Hong Kong RGC under Grant 9048123 (CityU 21211518) and Grant 9042820 (CityU 11219019), in part by the Natural Science Foundation of China under Grant 61871342 and Grant 61871434, and in part by the Basic Research General Program of Shenzhen Municipality under Grant JCYJ20190808183003968

Keywords

  • Compression
  • Deep learning
  • Depth estimation
  • Image-based rendering
  • Light field
  • Super resolution

Fingerprint

Dive into the research topics of 'Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion'. Together they form a unique fingerprint.

Cite this