1. [Publications](/publications) 2. Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation # Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation ![](/sites/default/files/styles/wide/public/publications/learning-rigidity_0.png?itok=aeUnvbBq) Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real world applications, a dynamic scene is commonly captured by a moving camera (i.e., panning, tilting or hand-held), increasing the task complexity because the scene is observed from different view points. The main challenge is the disambiguation of the camera motion from scene motion, which becomes more difficult as the amount of rigidity observed decreases, even with successful estimation of 2D image correspondences. Compared to other state-of-the-art 3D scene flow estimation methods, in this paper we propose to learn the rigidity of a scene in a supervised manner from a large collection of dynamic scene data, and directly infer a rigidity mask from two sequential images with depths. With the learned network, we show how we can effectively estimate camera motion and projected scene flow using computed 2D optical flow and the inferred rigidity mask. For training and testing the rigidity network, we also provide a new semi-synthetic dynamic scene dataset (synthetic foreground objects with a real background) and an evaluation split that accounts for the percentage of observed non-rigid pixels. Through our evaluation we show the proposed framework outperforms current state-of-the-art scene flow estimation methods in challenging dynamic scenes. ## Authors Zhaoyang Lv (Georgia Institute of Technology) Kihwan Kim (NVIDIA) Alejandro Troccoli (NVIDIA) Deqing Sun (NVIDIA) James M. Rehg (Georgia Institute of Technology) [Jan Kautz](/person/jan-kautz) ## Publication Date Sunday, September 9, 2018 ## Published in [European Conference on Computer Vision (ECCV 2018)](https://eccv2018.org/) ## Research Area [Computer Vision](/research-area/computer-vision) [Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence) [Robotics](/research-area/robotics) ## External Links [arXiv paper (PDF)](https://arxiv.org/pdf/1804.04259.pdf) [Zhaoyang Lv's homepage](https://www.cc.gatech.edu/~zlv30/) [Video (youtube)](https://www.youtube.com/watch?v=MnTHkOCY790) [REFRESH dataset (TBD)](https://www.cc.gatech.edu/~zlv30/) [Presentation PDF (GTC18)](http://on-demand.gputechconf.com/gtc/2018/presentation/s8798-learning-rigidity-in-dynamic-scenes-for-scene-flow-estimation-v2.pdf) ## Uploaded Files [Video (mp4)](https://research.nvidia.com/sites/default/files/pubs/2018-09_Learning-Rigidity-in/eccv18-rigidity-final_x264.mp4 "Open video in new window")60.41 MB ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .