Audio Flamingo 3
Advancing Audio Intelligence with Fully Open Large Audio Language Models

[Paper]    [Code]   [Website Demo]

Audio Flamingo 3 - 7B: [Gradio] [Checkpoints]

Audio Flamingo 3-Chat - 7B: [Gradio] [Checkpoints]

Audio Flamingo 3 - Datasets: [AudioSkills-XL] [LongAudio-XL] [AF-Think] [AF-Chat]

Authors: Arushi Goel^★, Sreyan Ghosh^★, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro

Posted: Zhifeng Kong

^★ Equal contribution, Alphabetical order

Overview

In this paper, we introduce Audio Flamingo 3, a fully open-source Large Audio Language Model (LALM) with state-of-the-art performance in audio understanding and reasoning across 20+ benchmarks. In addition, AF3 brings several novel capabilities, including multi-turn, multi-audio chat, on-demand thinking, voice-to-voice interaction, and long-context audio reasoning (up to 10 minutes).

• AF3 introduces key capabilities including: (i) long-context audio QA (extending beyond sounds and including speech), and (ii) flexible, on-demand thinking, enabling the model to generate concise, CoT-style reasoning steps when prompted.

• We also present AF3-Chat, a fine-tuned variant of AF3 designed for multi-turn, multi-audio chat and voice-to-voice interaction.

• We propose novelties in data curation, audio encoder representation learning, and training strategies. Being fully open, we release our code, training recipes, and 4 new datasets to promote research in this space.

Expert Reasoning and Long Audio Understanding

We propose AudioSkills-XL, a large-scale audio question-answering (AQA) dataset designed to develop (large) audio-language models on expert-level reasoning and problem-solving tasks over short audio clips (≤30 seconds). It expands upon the original AudioSkills collection (proposed in Audio Flamingo 2) by adding approximately 4.5 million new QA pairs, resulting in a total of ~10 million diverse examples. AudioSkills-XL focuses on seven primary skills for sounds and music:

• Temporal Reasoning: Understanding temporal relationships in audio (order, attribute changes, referring, grounding).
• Attribute Identification: Recognizing specific event properties (e.g., loudness, speaker gender).
• Counting: Quantifying occurrences of target sounds at varying difficulty levels.
• Contextual Sound Event Reasoning: Inferring the purpose or cause of a sound in its acoustic context.
• Contextual Speech Event Reasoning: Explaining spoken utterances in relation to surrounding sounds or dialogue.
• Information Extraction: Pulling out detailed facts, entities, or responses from audio content.
• General Reasoning: Addressing complex questions that combine multiple reasoning skills.

and 6 primary skills for speech:

• Sarcasm Identification: Inferring sarcasm from speech by analyzing content, tone, and emotional cues.
• Emotional State Reasoning: Identifying a speaker’s emotion, reasoning about its cause, and explaining any emotion flips.
• Topic Relationship Reasoning: Determining how two ideas or topics relate within the conversation.
• Information Extraction (IE): Needle QA, Causal QA, Response QA, and Topic QA for extracting specific facts, causes, responses, or main topics.
• Summarization: Producing a concise summary of the speech content.
• Order: Temporal Order, Temporal Attribute, Temporal Referring, and Temporal Grounding to locate and sequence topics over time.

We also propose LongAudio-XL a large-scale long audio question-answering (AQA) dataset designed to develop (large) audio-language models on long audio reasoning and problem-solving tasks over long audio clips (30 seconds - 10 mins). It expands upon the original LongAudio collection (proposed in Audio Flamingo 2) by adding approximately 1 million new QA pairs for long speech, resulting in a total of ~1.25 million diverse examples. LongAudio-XL focuses on six primary skills for sounds and music:

• Captioning: Generate comprehensive descriptions of long audio, capturing key events and the overall context.
• Plot QA: Answer questions about the audio’s narrative or storyline, reasoning over temporal and causal relationships.
• Temporal QA: Identify when events occur and how they relate in time, including sequencing, overlap, and attribute changes.
• Needle QA: Locate and reason about a specific “needle” segment within a longer audio “haystack,” ensuring answers reference that segment.
• Subscene QA: Answer questions about a distinct subscene in the audio, requiring focus on localized events and details.
• General QA: Address broad, open-ended questions spanning multiple events or themes, demonstrating overall comprehension.

and 6 primary skills for speech similar to AudioSkills-XL.

On-demand Thinking

We propose AF-Think, a novel dataset designed to develop on-demand thinking capabilities in audio-language models. It consists of 500K QA pairs, where each question is designed to require a chain-of-thought (CoT) reasoning process. The questions are carefully crafted by augmenting the original questions with a short thought prefix that precedes the answer and a special suffix to trigger thinking only when requested. AF-Think examples are sampled from both AudioSkills-XL and LongAudio-XL to cover diverse audio lengths and reasoning skills.

Multi-turn, Multi-audio Chat

We also propose AF-Chat which is a high-quality fine-tuning dataset of ~75K multi-turn, multi-audio conversations (avg. 4.6 clips & 6.2 turns; range 2–8 clips & 2–10 turns) spanning speech, environmental sounds, and music.

Samples

Emergent Audio Understanding

Audio Flamingo 3 understands audio with unusual sounds never seen during training, showing emergent capabilities.

Long audio and speech understanding

Below we show how Audio Flamingo 3 outperforms baselines on long audio and speech understanding, for audios up to 10 minutes in length.

Chain-of-Thought Reasoning

Audio Flamingo 3 supports flexible, on-demand reasoning, steering the model to correct predictions with accurate reasoning steps.

Short Audio Understanding

Audio Flamingo 3 not only achieves the state-of-the-art on audio understanding benchmarks, but also understands audio in depth.

Multi-audio Multi-turn Chat

Users can chat with Audio Flamingo 3, ask questions about multiple audios and refer to previous audios in context.

Sound Dialogue

Music Dialogue

Users interact with Audio Flamingo 3 about music, exploring genre blending, emotional tone, and production techniques across multiple clips.

Voice-to-voice conversation

Audio Flamingo 3 can respond to voice queries by speaking, supporting direct voice-to-voice interaction.

AF-Whisper

Prior work in audio representation learning typically treats speech, sounds, and music as separate modalities, and LALMs often rely on distinct encoders for each. Using separate encoders for LALMs increases model complexity, introduces frame-rate mismatches, and can lead to training instability. To address this, we propose AF-Whisper, a unified audio encoder trained with a simple yet effective representation learning strategy to model all three audio types.

Specifically, we use the pre-trained Whisper large-v3 encoder, attach it to a standard Transformer decoder, and train using the audio captioning task with the next-token-prediction objective. To achieve this, we generate a natural language caption for each audio, describing its speech, sound, and music content. We choose Whisper as the backbone due to its existing speech understanding capabilities and its dense, high-resolution audio features, which are more informative than those from models like CLAP.

Curriculum Training

Audio Flamingo 3 is trained with a 5-stage curriculum strategy as shown in the figure below.

Benchmark Results

Audio Flamingo 3 outperforms prior SOTA models including GAMA, Audio Flamingo, Audio Flamingo 2, Qwen-Audio, Qwen2-Audio, Qwen2.5-Omni, LTU, LTU-AS, SALMONN, AudioGPT, Gemini Flash v2, Gemini Pro v1.5, Gemini Pro v2.5, and GPT-4o-audio on a number of understanding and reasoning benchmarks.

• Audio Flamingo 3 has SOTA audio understanding and reasoning abilities, including long audio understanding.

• Audio Flamingo 3 achieves competitive/SOTA performance on the ASR benchmarks.

• Audio Flamingo 3 achieves SOTA performance on the multi-turn, multi-audio chat and voice-to-text benchmarks.

Citation

• Audio Flamingo

  @inproceedings{kong2024audio,
  title={Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities},
  author={Kong, Zhifeng and Goel, Arushi and Badlani, Rohan and Ping, Wei and Valle, Rafael and Catanzaro, Bryan},
  booktitle={International Conference on Machine Learning},
  pages={25125--25148},
  year={2024},
  organization={PMLR}
  }
  

• Audio Flamingo 2

  @inproceedings{
  ghosh2025audio,
  title={Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities},
  author={Ghosh, Sreyan and Kong, Zhifeng and Kumar, Sonal and Sakshi, S and Kim, Jaehyeon and Ping, Wei and Valle, Rafael and Manocha, Dinesh and Catanzaro, Bryan},
  booktitle={Forty-second International Conference on Machine Learning},
  year={2025},
  url={https://openreview.net/forum?id=xWu5qpDK6U}
  }
  

• Audio Flamingo 3

  @article{goel2025audio,
  title={Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models},
  author={Goel, Arushi and Ghosh, Sreyan and Kim, Jaehyeon and Kumar, Sonal and Kong, Zhifeng and Lee, Sang-gil and Yang, Chao-Han Huck and Duraiswami, Ramani and Manocha, Dinesh and Valle, Rafael and Catanzaro, Bryan},
  journal={arXiv preprint arXiv},
  year={2025}
  }