DreamStone: Image as a Stepping Stone for Text-Guided 3D Shape Generation

Zhengzhe LIU, Peng DAI, Ruihui LI, Xiaojuan QI, Chi-Wing FU

Research output: Journal PublicationsJournal Article (refereed)peer-review

Abstract

This paper presents a new text-guided 3D shape generation approach DreamStone that uses images as a stepping stone to bridge the gap between the text and shape modalities for generating 3D shapes without requiring paired text and 3D data. The core of our approach is a two-stage feature-space alignment strategy that leverages a pre-trained single-view reconstruction (SVR) model to map CLIP features to shapes: to begin with, map the CLIP image feature to the detail-rich 3D shape space of the SVR model, then map the CLIP text feature to the 3D shape space through encouraging the CLIP-consistency between the rendered images and the input text. Besides, to extend beyond the generative capability of the SVR model, we design the text-guided 3D shape stylization module that can enhance the output shapes with novel structures and textures. Further, we exploit pre-trained text-to-image diffusion models to enhance the generative diversity, fidelity, and stylization capability. Our approach is generic, flexible, and scalable. It can be easily integrated with various SVR models to expand the generative space and improve the generative fidelity. Extensive experimental results demonstrate that our approach outperforms the state-of-the-art methods in terms of generative quality and consistency with the input text.
Original languageEnglish
Pages (from-to)14385-14403
Number of pages19
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume45
Issue number12
Early online date2 Oct 2023
DOIs
Publication statusPublished - Dec 2023
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.

Keywords

  • 3D shape stylization
  • CLIP
  • score distillation sampling
  • text to 3d shape generation

Fingerprint

Dive into the research topics of 'DreamStone: Image as a Stepping Stone for Text-Guided 3D Shape Generation'. Together they form a unique fingerprint.

Cite this