1. [Publications](/publications)
2. Joint-task Self-supervised Learning for Temporal Correspondence
 
 # Joint-task Self-supervised Learning for Temporal Correspondence 

  ![](/sites/default/files/styles/wide/public/publications/nipsteser.png?itok=OalrBUD-)

 This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions and establishing fine-grained pixel-level associations between consecutive video frames. We exploit the synergy between both tasks through a shared inter-frame affinity matrix, which simultaneously models transitions between video frames at both the region- and pixel-levels. While region-level localization helps reduce ambiguities in fine-grained matching by narrowing down search regions; fine-grained matching provides bottom-up features to facilitate region-level localization. Our method outperforms the state-of-the-art self-supervised methods on a variety of visual correspondence tasks, including video-object and part-segmentation propagation, keypoint tracking, and object tracking. Our self-supervised method even surpasses the fully-supervised affinity feature representation obtained from a ResNet-18 pre-trained on the ImageNet.



 ## Authors



Xueting Li (UC Merced)

[Sifei Liu](/person/sifei-liu)

[Shalini De Mello](/person/shalini-de-mello)

Xiaolong Wang (UC Berkeley)

[Jan Kautz](/person/jan-kautz)

Ming-Hsuan Yang (UC Merced)

 

 

 ## Publication Date



Sunday, December 8, 2019

 

 ## Published in



[Neural Information Processing Systems (NeurIPS) 2019](https://sites.google.com/view/uvc2019)

 

 ## Research Area



[Computer Vision](/research-area/computer-vision)

 

 

 ## External Links



[paper](https://arxiv.org/pdf/1909.11895.pdf)

[Code](https://github.com/Liusifei/UVC)

[Video](https://youtu.be/Rfy3dG3lRpM)

 

 

 ## Copyright



This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to <pubs-permissions@ieee.org>.