Spatial Intelligence Lab
NVIDIA Research
Multi-view 3D reconstruction methods remain highly sensitive to photometric inconsistencies arising from camera optical characteristics and variations in image signal processing (ISP). Existing mitigation strategies such as per-frame latent variables or affine color corrections lack physical grounding and generalize poorly to novel views. We propose the Physically-Plausible ISP (PPISP) correction module, which disentangles camera-intrinsic and capture-dependent effects through physically based and interpretable transformations. A dedicated PPISP controller, trained on the input views, predicts ISP parameters for novel viewpoints, analogous to auto exposure and auto white balance in real cameras. This design enables realistic and fair evaluation on novel views without access to ground-truth images. PPISP achieves SoTA performance on standard benchmarks, while providing intuitive control and supporting the integration of metadata when available.
Our proposed pipeline applies a sequence of physically-grounded modules to the input reconstructed radiance (exposure offset, chromatic vignetting, linear color correction and non-linear camera response function).
Top row: During the first training phase, all modules except the controller are jointly optimized.
Bottom row: The controller is then trained to predict per-frame exposure and color correction for novel views while other modules are frozen.
In absence of an image formation model, a radiance field reconstruction will model photometric variations as geometric artifacts. PPISP improves the reconstruction quality by explaining away photometric variations in the input.
By disentangling the camera-specific vignetting and response function from other image formation effects, they generalize to novel views.
Drag the white bar to compare results.
Other methods such as bilateral grids optimize photometric compensation on training views only, which does not generalize to novel views. To address this, we introduce a controller module which predicts per-frame parameters on novel views.
The PPISP controller works analogously to auto exposure and auto white balance in conventional cameras. Given the rendered radiance image, it predicts exposure offset and color correction parameters.
The combination of photometric compensation and control improves the quality and likeness of rendered novel views compared to the ground truth significantly.
Drag the white bars to compare results.
Paper
Code
Dataset
BibTeX
TBD
|
|