Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos

Hanxue Liang^1,2* Jiawei Ren^1,3* Ashkan Mirzaei^1,4* Antonio Torralba^1,5 Ziwei Liu³ Igor Gilitschenski⁴ Sanja Fidler^1,4,6 Cengiz Oztireli² Huan Ling^1,4,6 Zan Gojcic^1† Jiahui Huang^1† ¹ NVIDIA ² University of Cambridge ³ Nanyang Technological University ⁴ University of Toronto ⁵ MIT ⁶ Vector Institute */†: Equal contribution/advising

Paper

What do we do?

BTimer: Given a monocular video as input, our method reconstructs a 3D Gaussian Splatting (3DGS) representation at any desired timestamp in a feed-forward fashion.

Technical Contributions

We present the first feed-forward reconstruction model for dynamic scenes using a bullet-time formulation;

BTimer reconstructs a bullet-time scene within 150ms while reaching state-of-the-art performance on both static and dynamic scene datasets, even compared with optimization-based approaches.

Method Framework

The model takes as input a sequence of context frames and their Plücker embeddings, along with the context timestamp and target (`bullet') timestamp embeddings. It then directly predicts the 3DGS representation at the bullet timestamp.

Results

Qualitative Results

Qualitative Results on SORA scenes. We pause the video at a frame and move the camera to make a bullet-time effect.

Qualitative Results on DAVIS dataset. Left is the input video, right is the video rendered from a novel camera trajectory.

Qualitative Results on DyCheck iPhone dataset. Left is the input video, right two are the videos rendered from novel camera trajectories.

Dynamic Novel View Synthesis Benchmark Results

Quantitative Comparison — Baseline results on NVIDIA Dynamic Scene Dataset.

Results on Static Scenes

Results on the Tanks & Temples static scenes benchmark.

Effect of the NTE Module

Citation

@article{liang2024btimer,
  title={Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos},
  author={Liang, Hanxue and Ren, Jiawei and Mirzaei, Ashkan and Torralba, Antonio and Liu, Ziwei and Gilitschenski, Igor and Fidler, Sanja and Oztireli, Cengiz and Ling, Huan and Gojcic, Zan and Huang, Jiahui},
  title={arXiv preprint arXiv:2412.03526},
  year={2024}
  }