POTF: Prompt-based Object-centric Tensorial Field

Recent advancements reconstructing a 3D scene, particularly Neural Radiance Field (NeRF), have significantly enhanced 3D scene synthesis through continuous implicit functions for volume rendering. However, real-world applications often require the reconstruction of specific regions or objects within a scene due to practical considerations such as computational efficiency and data storage limitations. To address these needs, we present Prompt-based Object-centric Tensorial Field (POTF), a novel approach that integrates user-directed segmentation via the Grounded Segment Anything Model (Grounded SAM) with efficient rendering capabilities utilizing tensor decomposition techniques. Our model allows precise segmentation based on image and text inputs, enabling dynamic and interactive refinement of the target regions. The tensor decomposition significantly improves reconstruction speed and quality by efficiently represent the 3D radiance fields. POTF prioritizes the rendering of user-specified objects, optimizing computational resources while maintaining high fidelity in critical areas. This method offers a practical and adaptive solution for 3D scene synthesis, catering to the specific needs of various real-world applications.

The user provides a text prompt specifying the object to be rendered from the scene. The trained Grounded-SAM model utilizes this prompt and input images to detect and define the regions of this specified object within the scene. We model the scene with only the specified region as a tensorial radiance field, leveraging tensor decomposition to effectively represent the scene. Finally, volume rendering is performed on the tensorial radiance field to produce high-quality 3D representations of the objects specified by the user, achieving efficient and precise rendering tailored to user demand.