High-fidelity 3D assets produced by NovelGS. It's designed for sparse-view reconstruction and operates in conjunction with various complementary tools, including text-to-image generation, and image-to-multiview modeling. This collaborative framework facilitates the generation of text-to-3D (bottom) and image-to-3D (center), as well as the reconstruction of real-world objects (top).

Abstract

We introduce NovelGS, a diffusion model for Gaussian Splatting (GS) given sparse-view images. Recent works leverage feed-forward networks to generate pixel-aligned Gaussians, which could be fast rendered. Unfortunately, the method was unable to produce satisfactory results for areas not covered by the input images due to the formulation of these methods. In contrast, we leverage the novel view denoising through a transformer-based network to generate 3D Gaussians. Specifically, by incorporating both conditional views and noisy target views, the network predicts pixel-aligned Gaussians for each view. During training, the rendered target and some additional views of the Gaussians are supervised. During inference, the target views are iteratively rendered and denoised from pure noise. Our approach demonstrates state-of-the-art performance in addressing the multi-view image reconstruction challenge. Due to generative modeling of unseen regions, NovelGS effectively reconstructs 3D objects with consistent and sharp textures. Experimental results on publicly available datasets indicate that NovelGS substantially surpasses existing image-to-3D frameworks, both qualitatively and quantitatively. We also demonstrate the potential of NovelGS in generative tasks, such as text-to-3D and image-to3D, by integrating it with existing multiview diffusion models. The code is publicly available on GitHub.

Method

Pipeline of NovelGS model. We utilize a large transformer-based network to denoise noisy view images for 3D reconstruction. During inference, we initialize target views with pure noise. Then we concatenate the camera ray embedding (Plucker rays) and images ¨ (two clean views and one noisy view in the figure to reduce clutterness; four clean views and one noisy view in main experiments) as the input. Then we utilize the denoiser to predict the Gaussians and render the image from the noisy view. After that, we add noise to the noisy view images to timestep T-1. We loop this process until we get the final 3D Gaussians. During training, we add noise to the noisy view images based on the timestep and utilize the denoiser to predict 3D Gaussians. We train the denoiser module by rendering loss.