1. [Publications](/index.php/publications) 2. Context-aware Captions from Context-agnostic Supervision # Context-aware Captions from Context-agnostic Supervision ![](/sites/default/files/styles/wide/public/pubs/2019-04_Context-aware-Captions-from/default.jpg?itok=xyyLP3nX) We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of "siamese cat" and "tiger cat", we generate language that describes the "siamese cat" in a way that distinguishes it from "tiger cat". Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination ## Authors Ramakrishna Vedantam (Georgia Tech) Samy Bengio (Google) Kevin Murphy (Google) Devi Parikh (Georgia Tech ) [Gal Chechik](/index.php/person/gal-chechik) ## Publication Date Saturday, April 15, 2017 ## Published in [Computer Vision and Pattern Recognition](https://arxiv.org/abs/1701.02870) ## Research Area [Computer Vision](/index.php/research-area/computer-vision) [Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence) ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .