Spatial Intelligence Lab
Holoscan Healthcare
Reconstructions of surgical scenes from endoscopic video are useful for surgical training and medical animation, as they accurately capture the clinician's viewpoint. In practice, however, 3D scenes acquired through endoscopy, increasingly represented as 3D Gaussian splats (3DGS), are typically constrained to limited viewpoint ranges. Consequently, even small endoscope camera motions can significantly degrade rendering fidelity and organ shape reconstruction. This gap prevents the direct use of such scenes in physics-based surgical simulation, which requires accurate, shape-consistent geometry. To address this, we propose a pipeline that combines the visual realism of an endoscopic 3DGS scene with simulation-ready organ geometry extracted from pre-operative 3D computed tomography (CT) or MRI scans. Because the pre-operative and endoscopically observed organs often differ in pose and deformation and geometric completeness, we register the full segmented anatomy onto its partial corresponding region in the splat representation using a mesh-free, elastically regularized alignment method. Elastic deformation is represented using Simplicits, a mesh-free, reduced-order model based on learned skinning weights, allowing us to optimize on the order of 10–20 handle transforms rather than thousands of tetrahedral-mesh vertices. We employ a two-stage procedure: (1) a weighted rigid iterative closest point (ICP), (2) followed by a reduced-order elastic ICP, with an elastic energy term and geometry matching term that uses nearest-neighbor correspondences to the 3DGS target. The result is a simulation-ready scene where the deformation geometry is anatomically plausible and aligned to the real 3DGS visible scene.
OrganPhys requires two inputs—an intra-operative endoscopic video and a pre-operative CT scan. G-SHARP reconstructs the endo- scopic scene as a 3D Gaussian Splat whose Gaussian centres form the target point cloud (rendered in red in subsequent panels), while the CT volume is segmented, normalized, and tagged with anatomical landmarks to yield the source cloud (blue). A two-stage rigid cascade—coarse landmark alignment (Section 3.3) followed by weighted rigid ICP (Section 3.4)—deforming the full source organ into complete correspon- dence with the partial, endoscope-visible target. A reduced non-rigid stage then constructs Simplicits deformation bases (Section 3.5) and solves an elastic ICP (Section 3.6) in the resulting low-dimensional modal subspace, yielding a physics-consistent, simulation-ready organ registered to the endoscopic scene and ready for downstream interactive simulation. Dashed boxes denote off-the-shelf components; solid boxes mark our contributions.
For each organ we show the registered scene rendered with 3DGRUT alongside the underlying physics-based simulation in Polyscope. Select a tab above to switch between organs.
Nick Sharp for useful discussions and feedback. Jonathan Leaf for help in experimenting with Scan2Twin registration.