VideoFDB —
Evaluating Full-Duplex Vision-Speech
Capabilities in Conversational Agents

Natural human conversation is full-duplex and audio-visual: people simultaneously speak, listen, and signal through gaze, gesture, and affect. VideoFDB is the first benchmark to evaluate full-duplex audio-visual-to-audio-visual conversational agents — and we find that today's vision-speech models systematically miss the nonverbal turn.

Existing full-duplex benchmarks evaluate speech alone, while audio-visual benchmarks evaluate split-role or turn-based interaction. Neither captures the continuous, overlapping co-construction of meaning that defines natural dyadic conversation. VideoFDB closes that gap with 237 dyadic clips from real video calls spanning 11 nonverbal conversational dynamics, a taxonomy separating perception from generation, and a rubric-based LM-as-judge framework that scores agents along interpretable axes.

Evaluating leading open- and closed-source vision-speech agents, we find two systematic failure modes: captioning collapse (the model describes the user's appearance rather than conversing with them) and visual-stream ignorance (the audio-only and audio-visual outputs are paraphrases of each other). Cascaded speech-to-avatar pipelines preserve turn-yielding discipline but cannot insert nonverbal cues during the user's turn, with latencies 2.8–3.5 seconds behind human ground truth.

• A benchmark grounded in human communication science. 237 expert-annotated dyadic clips from natural video calls, spanning 11 conversational dynamics drawn from established interpersonal-communication research — turn-taking, backchannels, gaze aversion, adaptors, affect displays, and more.
• A perception/generation taxonomy with interpretable rubrics. Perception axes score whether the agent reads the situation; generation axes score whether the agent's own audio-visual output coheres. Three rubric axes per category, each scored 0–5 by an LM judge with stable cross-judge agreement (77–89% within 1 point).
• Next steps for research: We identify captioning collapse, visual-stream ignorance, and the lack of full-duplex avatar pipelines as key opportunities for future work in Interactive Agentic AI.

Consider a brief pause in the middle of a sentence (Figure 1). An audio-only agent may treat it as a turn handoff and start speaking. But with both audio and video together, the same moment has more context: a shifted gaze and raised head can signal the user is still thinking, so the right response is to wait.

What an agent does while the user is still speaking matters as much as what it says next and when. Most evaluations split dialogue into turns and focus on the response latency or audio quality, but real conversations are continuous, with bidirectional verbal and nonverbal cues. Full-duplex speech benchmarks measure performance from audio alone and would interrupt the user's pause. Video QA benchmarks reward identifying gaze direction, but not using it to guide turn-taking. VideoFDB evaluates the intersection between these two approaches: whether agents continuously use visual signals during two-way conversation.

We organize the benchmark around the four nonverbal channels human-communication research identifies as central to dyadic interaction — dialogue, eye gaze, face, and body — and select dynamics that directly govern conversational floor management, listener feedback, social-affective signaling, and conversational body movement.

Each clip is centered on a human-annotated dynamic event window [t_start, t_end], with 1–3 lead-in turns and up to one follow-up turn preserved as context. Clips are mined from a held-out corpus of two-person video calls and validated through a three-pass annotation pipeline (candidate discovery → timestamp/type validation → final quality review).

@misc{mazumdar2026videofdb,
      title         = {VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents},
      author        = {Mazumdar, Amrita and Park, Seonwook and Roy, Rajarshi and Srihari, Nikhil and Wang, Shengze and Zhou, Yuhao and Wang, Julia and Nagano, Koki and De Mello, Shalini},
      year          = {2026},
      eprint        = {2605.30256},
      archivePrefix = {arXiv},
      primaryClass  = {cs.CV},
      url           = {https://arxiv.org/abs/2605.30256}
}