SRInpaintor : When Super-Resolution Meets Transformer for Image Inpainting

Feng LI, Anqi LI, Jia QIN, Huihui BAI*, Weisi LIN, Runmin CONG, Yao ZHAO

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

4 Citations (Scopus)

Abstract

Recent image inpainting methods have achieved remarkable improvements by using generative adversarial networks (GAN). Most of them have been designed to produce plausible results from high-level semantic features using only high-resolution (HR) supervision. However, because abundant details are lost in large holes, it is difficult to simultaneously synthesize details while preserving structural coherence in HR space. Besides, the correlations between the inside and outside of the missing region play a critical role in transferring relevant known information to generate semantic-coherent textures, especially in patch matching-based methods. In this work, we present SRInpaintor which inherits the merits of super-resolution (SR) and transformer for high-fidelity image inpainting. The SRInpaintor starts from global structure reasoning with low-resolution (LR) input and progressively refines the local textures in HR space, constituting a multi-stage framework with SR supervision. The bottom stage recovers coarse SR results that provide structural information as an appearance prior, and is combined with the higher-resolution corrupted image at the next stage to render available textures for the missing region. Such a design can analyse the image from LR to HR with the increase of stages, enabling coarse-to-fine information propagation and detail refinement. In addition, we propose a hierarchical transformer (HieFormer) to model the long-term correlations between distant contexts and holes. By embedding it into a compact latent space in a cross-scale manner, we can ensure reliable relevant texture transformation and robust appearance consistency. Experimental results demonstrate the superiority of our method compared with recent state-of-the-art methods. Code will be available on https://github.com/lifengshiwo/SRInpaintor.

Original languageEnglish
Pages (from-to)743-758
Number of pages16
JournalIEEE Transactions on Computational Imaging
Volume8
Early online date12 Jul 2022
DOIs
Publication statusPublished - 2022
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2015 IEEE.

Keywords

  • Generative image inpainting
  • progressive super-resolution
  • transformer

Fingerprint

Dive into the research topics of 'SRInpaintor : When Super-Resolution Meets Transformer for Image Inpainting'. Together they form a unique fingerprint.

Cite this