Learning Physically Simulated Tennis Skills from Broadcast Videos


We present a system that learns diverse, physically simulated tennis skills from large-scale demonstrations of tennis play harvested from broadcast videos. Our approach is built upon hierarchical models, combining a low-level imitation policy and a high-level motion planning policy to steer the character in a motion embedding learned from broadcast videos. When deployed at scale on large video collections that encompass a vast set of examples of real-world tennis play, our approach can learn complex tennis shotmaking skills and realistically chain together multiple shots into extended rallies, using only simple rewards and without explicit annotations of stroke types. To address the low quality of motions extracted from broadcast videos, we correct estimated motion with physics-based imitation, and use a hybrid control policy that overrides erroneous aspects of the learned motion embedding with corrections predicted by the high-level policy. We demonstrate that our system produces controllers for physically-simulated tennis players that can hit the incoming ball to target positions accurately using a diverse array of strokes (serves, forehands, and backhands), spins (topspins and slices), and playing styles (one/two-handed backhands, left/right-handed play). Overall, our system can synthesize two physically simulated characters playing extended tennis rallies with simulated racket and ball dynamics.


Diverse Tennis Skills


Forehand topspin

Backhand topspin

Backhand slice




Task Performance

Same incoming ball + Different target

Different incoming ball + Same target

Different Player Styles

Right-hand, two-handed backhand

Left-hand, two-handed backhand

Two Player Rallies

Right-hand, one-handed backhand (near) vs. Right-hand, two-handed backhand (far)

View 1

View 2

View 3

Left-hand, two-handed backhand (near) vs. Right-hand, one-handed backhand (far)

View 1

View 2

View 3


  author = {Zhang, Haotian and Yuan, Ye and Makoviychuk, Viktor and Guo, Yunrong and Fidler, Sanja and Peng, Xue Bin and Fatahalian, Kayvon},
  title = {Learning Physically Simulated Tennis Skills from Broadcast Videos},
  journal = {ACM Trans. Graph.},
  issue_date = {August 2023},
  numpages = {14},
  doi = {10.1145/3592408},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {physics-based character animation, imitation learning, reinforcement learning},


All the broadcast videos are provided courtesy of the Tennis Channel.


For any question regarding this research paper, please contact Haotian Zhang via haotianz@cs.stanford.edu.

Template adapted from GLAMR.