Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning

Berger, Uri; Chechik, Gal; Dalal, Gal

Look Where You Say You’re Looking

Self-Grounded Attention for Visual Reasoning

Uri Berger^1,2,3, Gal Chechik^1,4, Gal Dalal¹

¹NVIDIA Research ²University of Melbourne ³The Hebrew University of Jerusalem ⁴Bar-Ilan University

PaperarXiv:2610.05023
CodeComing soon
WeightsComing soon
BibTeX

TL;DR

We introduce Self-Saliency, a method for aligning where a model looks with what it talks about. Self-Saliency uses a grounding model to localize the objects mentioned in each reasoning step, and treats the resulting regions as supervision for the model's visual attention. Across a broad suite of 25 visual reasoning benchmarks, Self-Saliency surpasses all previous baselines and outperforms the vanilla model by 3.2 points on average.

Method

Pipeline diagram: an input image and reasoning step feed a grounding model that produces bounding boxes; the model's visual attention produces a saliency map; the overlap between the two gives a saliency score, which joins the format and accuracy rewards as input to GRPO. — **The saliency-scoring pipeline.** Given an input image and reasoning step, we extract bounding boxes corresponding to the mentioned image regions, compute a saliency map from the model's visual attention, and calculate the saliency score from the overlap between the two. The saliency score is used as one reward element in GRPO, in addition to accuracy and format elements.

Results

Benchmark	Qwen3-VL 8B-Instruct	+SFT	+Vanilla GRPO	VGA	Saliency-R1	EASE	Self-Saliency
	No attention intervention			Prior work			Ours
Mean score ↑	61.1	62.7	63.5	61.2	63.6	62.9	64.3
Mean rank ↓	4.42	4.64	3.68	4.48	3.26	4.50	3.02
Wins ↑	4	0	3	3	6	0	9
HR-Bench 8K	74.1	72.9	72.0	74.2	76.0	72.8	74.5
DailyClue	35.4	32.4	34.2	34.2	32.0	32.7	31.2
ChartQA	85.0	88.8	89.4	84.9	89.7	88.9	90.0
CV-Bench	86.0	83.4	84.9	86.1	83.8	84.2	85.9
HallusionBench	63.2	59.7	57.1	62.0	58.3	62.6	55.7
MathVision	29.6	33.5	37.5	31.2	33.9	35.9	41.5
MME	2375.9	2371.7	2376.1	2379.8	2395.7	2395.4	2355.1
MMK12	68.2	63.1	62.3	67.9	61.1	63.1	62.0
HR-Bench 4K	78.0	76.9	77.4	78.1	78.9	76.9	79.5
MathVerse	34.4	51.0	52.8	33.8	49.5	50.4	50.9
SalBench	43.3	60.7	59.8	43.3	62.7	58.6	64.7
MathVista	73.5	69.5	71.8	73.8	71.0	70.0	70.4
LogicVista	52.5	54.9	55.1	51.3	56.5	52.9	56.2
MME-RealWorld	63.7	64.2	66.7	63.8	66.4	65.1	68.7
IllusionVQA	40.0	41.9	41.5	40.1	43.8	40.5	43.5
MMMU-Pro	40.4	48.6	48.6	41.9	47.7	47.1	48.4
AlgoPuzzleVQA	32.8	36.9	39.9	34.3	38.7	36.4	41.4
RealWorldQA	68.5	67.1	67.8	68.4	68.8	67.5	68.2
OmniSpatial	47.4	50.3	48.2	46.8	50.2	50.2	51.0
V*	84.3	79.6	79.6	85.3	80.6	81.2	82.7
WeMath	69.4	68.6	74.3	68.2	73.8	71.1	75.5
ScienceQA-IMG	94.4	96.3	97.2	94.4	97.6	96.2	97.4
VisuLogic	26.6	24.9	24.8	26.4	24.9	24.4	24.4
POPE	88.8	87.4	88.2	88.8	88.5	88.5	88.9
MMStar	64.0	69.8	70.9	64.7	69.9	69.5	69.6

Results across 25 benchmarks. The largest gains are on benchmarks that need low-level visual saliency and multi-step reasoning (AlgoPuzzleVQA, SalBench, MME-RealWorld). Surprisingly, we also see notable gains on mathematical reasoning (WeMath, MathVision), where the grounding model was never trained to operate.

Qualitative examples in which Self-Saliency attends to and correctly reads small, low-level visual details. — **Qualitative examples.**

Analysis

Training changes the text at the selected heads, and the attention across all of layer 22

We asked how training raises the reward: (1) by changing where visual attention goes, or (2) by changing the generated text.

Text. The model generates shorter completions, which refer to larger portions of the image.

Attention. When we trained Self-Saliency, we computed the reward on two selected heads (layer 22, heads 28 and 31). On the selected heads, the fraction of attention in referred regions stays the same after training. Across all the heads of layer 22, however, that fraction increases significantly.

	Completion length *	Referred portion of image **	Fraction of attention in referred regions (selected heads)	Fraction of attention in referred regions (mean across layer-22 heads) ***
	Text-related		Attention-related
Vanilla	230.0	0.533	0.462	0.611
Self-Saliency	149.6	0.566	0.462	0.622

Text and attention statistics before and after training, averaged over 100 held-out validation samples. * p ≪ 0.00001, Wilcoxon signed-rank test
** p = 0.00009, Wilcoxon signed-rank test
*** p ≪ 0.00001, Wilcoxon signed-rank test

Attention piles up on the image border

Across VLMs, visual attention concentrates on the geometric border of the image. Moreover, the most attended patch sits in a corner. In contrast, human annotators marked the center as the important region of the image.

Five patch-grid heatmaps, separated by a vertical rule. The four model maps on the left light up along the image border with a bright corner; the human-annotation map on the right is brightest in the middle. — **Saliency maps for different models and human-annotated boxes.** For models (left) we compute saliency using attention scores. For humans (right) we compute saliency using annotated boxes.

Two patch-grid heatmaps of the two heads used to compute the reward, before and after training. Before training the top-left corner is violet, the colour reserved for patches above 16 times a fair share. After training the same corner is red, inside the scale, and the attention is spread along the border and toward the center. — **The two heads used to compute the reward, before and after training.** Attention is redistributed primarily from the corner sinks to the image border, with a smaller fraction moving toward the center.

Ablations

We run two ablations.

One explanation for Self-Saliency's gain is that it merely shifts attention toward the center of the image. To rule this out, we replace the grounding model's boxes with a fixed rectangle at the center, and train on that. We name this variant Center rect.

Another explanation is that response-specific grounding does not matter. Under this hypothesis, any boxes relevant to the image supply the same supervision. To rule this out, we give the grounding model the input question, and use the resulting boxes as supervision. We name this variant Question boxes.

Self-Saliency beats both ablations. Grounding the model's own response contributes meaningful feedback for training.

	Qwen3-VL-8B Instruct	+SFT	+Vanilla GRPO	Center rect	Question boxes	Self-Saliency
Mean score ↑	61.14	62.69	63.47	63.39	63.33	64.26
Mean rank ↓	3.90	4.06	3.34	3.68	3.38	2.64
Wins ↑	7	0	3	2	5	9

Ablations across the same 25-benchmark suite. Grounding the model's own reasoning chain provides a meaningfully better training signal than either a fixed central rectangle or the grounded question text.

BibTeX

@article{berger2026selfsaliency,
  title   = {Look Where You Say You're Looking:
             Self-Grounded Attention for Visual Reasoning},
  author  = {Berger, Uri and Chechik, Gal and Dalal, Gal},
  journal = {arXiv preprint arXiv:2610.05023},
  year    = {2026}
}