1. [Publications](/publications)
2. Convolutional Tensor-Train LSTM for Spatio-Temporal Learning
 
 # Convolutional Tensor-Train LSTM for Spatio-Temporal Learning

  ![Publication image](/sites/default/files/styles/wide/public/default_images/default.jpeg?itok=qUFsuJCP "Publication image")

 Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation. However, existing methods still perform poorly on challenging video tasks suchas long-term forecasting. The gap partially is because these kinds of challenging tasks require learning long-term spatio-temporal correlations in the video sequence. We propose a higher-order convolutional LSTM model that can efficiently learn these correlations with a succinct representation of the history. Our model relies on a novel tensor-train module that performs prediction by combining convolutional features across time. To make computation and memory requirements feasible, we develop a novel convolutional tensor-train decomposition of the higher-ordermodel. This decomposition reduces the model complexity by jointly approximatinga sequence of convolutional kernels as a low-rank tensor-train factorization. Asa result, our model outperforms existing approaches but uses only a fraction of parameters, including the baseline models. Our results achieve state-of-the-artperformance in a wide range of applications and datasets, including the multi-stepsvideo prediction on the Moving-MNIST-2 and KTH action datasets as well as earlyactivity recognition on the Something-Something V2 dataset.



 ## Authors



Jiahao Su (University of Maryland)

[Wonmin Byeon](/person/wonmin-byeon)

[Jean Kossaifi](/person/jean-kossaifi)

Furong Huang (University of Maryland)

[Jan Kautz](/person/jan-kautz)

Anima Anandkumar (NVIDIA)

 

 

 ## Publication Date



Sunday, December 6, 2020

 

 ## Published in



[Advances in Neural Information Processing Systems (NeurIPS)](https://papers.nips.cc/paper/2020/hash/9e1a36515d6704d7eb7a30d783400e5d-Abstract.html)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

 

 

 ## External Links



[Paper](https://papers.nips.cc/paper/2020/hash/9e1a36515d6704d7eb7a30d783400e5d-Abstract.html)

[Code](https://github.com/NVlabs/conv-tt-lstm)

[Slides](https://docs.google.com/presentation/d/1wnjgvaR5jixWx5RYwHu9Ne4rt1gsmmXepygFOTt2914/edit#slide=id.p1)