1. [Publications](/index.php/publications)
2. Contrastive Learning for Weakly Supervised Phrase Grounding
 
 # Contrastive Learning for Weakly Supervised Phrase Grounding

  ![](/sites/default/files/styles/wide/public/publications/contrastive.png?itok=GWYNjrmE)

 Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ~10% absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7% to achieve 76.7% accuracy on Flickr30K Entities benchmark.



 ## Authors



Tanmay Gupta

[Arash Vahdat](/index.php/person/arash-vahdat)

[Gal Chechik](/index.php/person/gal-chechik)

Xiaodong Yang

[Jan Kautz](/index.php/person/jan-kautz)

Derek Hoiem

 

 

 ## Publication Date



Wednesday, June 17, 2020

 

 ## Published in



[European Conference on Computer Vision (ECCV) 2020 (Spotlight)](https://arxiv.org/abs/2006.09920)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence)

[Computer Vision](/index.php/research-area/computer-vision)