Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Jing JIN, Junhui HOU, Jie CHEN, Sam KWONG, Jingyi YU

Research output: Book Chapters | Papers in Conference ProceedingsConference paper (refereed)Researchpeer-review

18 Citations (Scopus)

Abstract

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission. The code is available at https://github.com/jingjin25/LFhybridSR-Fusion.
Original languageEnglish
Title of host publicationMM '20: Proceedings of the 28th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery
Pages193-201
Number of pages9
ISBN (Print)9781450379885
DOIs
Publication statusPublished - Oct 2020
Externally publishedYes
Event28th ACM International Conference on Multimedia (MM 2020) - Virtual, Seattle, United States
Duration: 12 Oct 202016 Oct 2020
https://2020.acmmm.org/

Conference

Conference28th ACM International Conference on Multimedia (MM 2020)
Country/TerritoryUnited States
CitySeattle
Period12/10/2016/10/20
Internet address

Bibliographical note

This work was supported in part by the Hong Kong RGC under Grant 9048123 (CityU 21211518), and in part by the Basic Research General Program of Shenzhen Municipality under Grant JCYJ20190808183003968.

Keywords

  • attention
  • deep learning
  • hybrid imaging system
  • light field
  • super-resolution

Fingerprint

Dive into the research topics of 'Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses'. Together they form a unique fingerprint.

Cite this