ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop

Abstract

Representation in the family of 3D Gaussian Splats (3DGS) are growing into a viable alternative to traditional graphics for an expanding number of application, including recent techniques that facilitate physics simulation and animation. However, extracting usable objects from in-the-wild captures remains challenging and controllable editing techniques for this representation are limited. Unlike the bulk of emerging techniques, focused on automatic solutions or high-level editing, we introduce an interactive suite of tools centered around versatile Gaussian Splat selection and segmentation. We propose a fast AI-driven method to propagate user-guided 2D selection masks to 3DGS selections. This technique allows for user intervention in the case of errors and is further coupled with flexible manual selection and segmentation tools. These allow a user to achieve virtually any binary segmentation of an unstructured 3DGS scene. We evaluate our toolset against the state-of-the-art for Gaussian Splat selection and demonstrate their utility for downstream applications by developing a user-guided local editing approach, leveraging a custom Video Diffusion Model. With flexible selection tools, users have direct control over the areas that the AI can modify. Our selection and editing tools can be used for any in-the-wild capture without additional optimization.

Method

Imagine working with a monolithic capture of a cluttered environment, like a play room. To construct an interactive environment, this scene must first be broken apart into individual objects. This is the core task addressed by our method, showing why fully automatic solutions do not work perfectly, and how the user could more directly control the final output. Flexible selection forms the necessary backbone for many applications.

(Above): The clean and consistent design of 2D and 3D selection in our toolkit combines automatic AI-driven techniques with user input and could be integrated into future end-user applications for crafting interactive 3DGS scenes.

(Below): Any implementation of our proposed toolkit will allow users control of the camera view and will keep track of the 2D segmentation mask (in blue, active for the camera view) and of the current 3D segmentation mask (in yellow), containing a binary value for every Gaussian in the scene. We propose combining multiple ways to project 2D masks to the 3D segmentation mask, we are showing manual projection modes and also an automatic mask tracking and segmentation method.

Results

Here are a few examples of real-world usage, showing the efficiency of our automatic method and the pertinence of our manual tools for fine-tuning the selection.

Applications

(Left): Targeted object completion and editing. (Right): Physics Simulation, using Simplicits method.

Citation

    @article{fujitsang2026artisangs,
        title = {ArtisanGS: Interactive Tools for Gaussian Splat Selection with AI and Human in the Loop},
        author = {Clement {Fuji Tsang} and
                  Anita {Hu} and
                  Or {Perel} and
                  Carsten {Kolve} and
                  Maria Shugrina},
        journal={arXiv preprint arXiv:2602.10173},
        year={2026},
    }