NVIDIA

NVIDIA

NVIDIA Nemotron 3.5 Transcribe: Intelligent Transcription with a Speech LLM

Published:

Technical Report    Model Weights 🤗    License

Overview

There is no single correct transcript of a recording: what counts as correct depends on who reads it and for what purpose. A transcript may need numbers in spoken or written form, verbatim wording or cleaned-up disfluencies, a particular notation for domain-specific terms, or a specific script for code-switched speech. Conventional speech recognition systems commit to one such convention when they are trained.

We introduce Nemotron 3.5 Transcribe, a speech-augmented language model (SALM) built for this kind of intelligent transcription. It couples a Transformer speech encoder and a frozen Nemotron 3 Diarization model with the Nemotron 3.5 Lightning 30B-A3B hybrid Mamba–Transformer mixture-of-experts LLM, for approximately 4.3B active parameters. A single model performs verbatim, written-form, contextual, speaker-attributed, and rich transcription, speech translation, and spoken language identification in 40 languages, on recordings up to one hour long, and returns speaker-attributed word-level timestamps on request. The transcription style is chosen with a natural language instruction for each request, without retraining.

Nemotron 3.5 Transcribe Overview

Highlights

  • Best open-weights model on the Open ASR Leaderboards: a mean WER of 4.71% on the 9 public test sets of the English Open ASR Leaderboard and 4.62% on the multilingual Open ASR Leaderboard.
  • Speaker-attributed transcription: state-of-the-art accuracy on speaker-attributed ASR for overlapping multi-speaker speech, from one-minute segments to full meetings and calls. To our knowledge, it is the first SALM to outperform specialized cascaded systems on this task.
  • Contextual transcription: given a list of expected entities, Nemotron 3.5 Transcribe reduces named-entity errors on ContextASR-Bench six-fold.
  • Word timestamps: a lightweight CTC adapter aligns any generated transcript to the audio after decoding, which yields speaker-attributed word timestamps for long and heavily overlapping recordings.
  • Efficient inference: native Multi-Token Prediction (MTP) speculative decoding and an optional DFlash2 draft model. Served by the Nemotron Transcribe NIM, the NVFP4 checkpoint with DFlash2 transcribes up to 2,238 seconds of audio per second on a single RTX PRO 6000 Blackwell GPU.

Serving Performance

Technical Contributions

Architecture

  • Parallel Expert Encoder: speaker activity from a frozen diarization model is injected into the ASR encoder representations, which enables speaker-attributed transcription with serialized output training without architectural changes to the LLM.
  • Supervision-balanced training mixture: training data is sampled by its answer-token mass, and every ASR example can be used with one of many task instructions. This adds capabilities with sub-percent sampling rates, in a single supervised fine-tuning stage followed by a short reinforcement learning stage that optimizes transcription accuracy.
  • Joint MTP training: the model is trained jointly with a multi-token prediction objective, so its own MTP layer serves as the draft model for speculative decoding in vLLM; a separately trained DFlash2 draft model is provided as an alternative.
  • Practical prompting and fine-tuning: we list the exact instructions used in our evaluation as prompting recommendations, and show on 17 tasks that the model can be adapted to new languages, domains, and tasks on a single GPU, typically within 0.5–2.3 GPU-hours per task.

Open Source

We release the model under the OpenMDW-1.1 license, together with a fine-tuning toolkit in NeMo Speech.

Model checkpoints

Citation

@techreport{Nemotron_3_5_Transcribe,
  title={Nemotron 3.5 Transcribe Technical Report},
  author={NVIDIA},
  institution={NVIDIA},
  year={2026}
}