QueryWhat is the child wearing?
Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure.
We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector.
We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55% at 1.06× the latency of standard generation.
One forward pass of a frozen LVLM yields a heterogeneous computational trace. UniProbe reads it as structured data instead of flattening it into a feature vector.
Nodes are tokens of three types: all T response tokens, plus the image patches and query tokens that receive the most total attention from the response. Each node carries its layer-ℓ hidden state; image nodes additionally carry a 2-D encoding of their patch coordinates. Edges come straight from attention — for each response token we keep the strongest incoming connections from image patches, query tokens, and earlier response tokens. By causality, the neighborhood of a token is exactly the visual and textual evidence the model used to emit it.
A GNN mixes evidence across modalities with type-specific projections, so “attends to an image patch” and “attends to the query” read as distinct evidence. A ViT then refines the image grid and a BiGRU the response in generation order. Interleaving the three — rather than running them in parallel — lets spatial, relational, and sequential evidence interact throughout the network. The whole detector is trained end-to-end while the backbone stays frozen.
Replacing the bidirectional response GRU with a unidirectional one makes the detector streaming: the score of token ri depends only on the prefix r≤i. As the model decodes, each new token is scored against the causal trace; if its hallucination probability exceeds τ, the token is rejected, its first sub-token banned, and the model re-decodes. The hallucination is corrected during generation and never reaches the final response.
Token-level benchmarks supervise detectors on responses from other LVLMs, but at deployment a detector must judge its host model's own, more fluent generations. We close this gap without manual annotation: caption a small set of Objects365 images with the target model, auto-label object mentions with CHAIR against ground-truth objects, and fine-tune UniProbe on those generations.
Two questions, one detector: which tokens of a response are unsupported (MHALO, HalLoc), and whether a mentioned object is actually in the image (self-generated COCO captions, POPE).
The same detector, made causal, runs inside the decoding loop: it scores each token the moment it is emitted, rejects the ones it flags, and re-decodes. Hallucinations are repaired mid-generation rather than after the fact.
Caption quality holds up (SPICE 0.222 vs. 0.214 vanilla) and the output distribution barely moves (JS divergence 0.013) — the guardrail removes hallucinations without rewriting the model.
Detection failures of prior methods, and what UniProbe generates instead.
If you find this work useful, please cite:
@article{samuel2026uniprobe,
title = {UniProbe: A Learnable Token-Level Hallucination Detector
for Large VLMs using Multi-Structural Internal Representations},
author = {Samuel, Dvir and Bar-Shalom, Guy and Frasca, Fabrizio and
Fetaya, Ethan and Ziser, Yftah and Chechik, Gal and
Maron, Haggai},
journal = {arXiv preprint arXiv:2608.10835},
year = {2026}
}