NVIDIA ADLR

NVIDIA ADLR

Projects

Audio Flamingo 2

Published:

Understanding and reasoning over non-speech sounds and music are crucial for both humans and AI agents to interact effectively with their environments. In this paper, we introduce Audio Flamingo 2, an Audio-Language Model (ALM) with advanced long-audio understanding and reasoning capabilities. Audio Flamingo 2 achieves the state-of-the-art performance across over 20 benchmarks, with only a 3B parameter small language model. In addition, we propose training and test sets for long audio understanding capabilities – namely LongAudio and LongAudioBench – to advance this field, and Audio Flamingo 2 is the first ALM that can understand long audio up to 5 minutes. We confirm the efficacy of our method through extensive evaluations and ablation studies.

A2SB: Audio-to-Audio Schrodinger Bridges

Published:

Audio in the real world may be perturbed due to numerous factors, causing the audio quality to be degraded. The following work presents an audio restoration model tailored for high-res music at 44.1kHz. Our model, Audio-to-Audio Schrodinger Bridges (A2SB), is capable of both bandwidth extension (predicting high-frequency components) and inpainting (re-generating missing segments). Critically, A2SB is end-to-end without need of a vocoder to predict waveform outputs, able to restore hour-long audio inputs, and trained on permissively licensed music data. A2SB is capable of achieving state-of-the-art bandwidth extension and inpainting quality on several out-of-distribution music test sets.

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

Published:

We introduce AceMath, a family of frontier math reasoning models that set new state-of-the-art performance on math reasoning benchmarks. AceMath outperforms both leading open-access models (e.g., Qwen2.5-Math-72B-Instruct) and proprietary models (e.g., GPT-4o (2024-08-06) and Claude 3.5 Sonnet (2024-10-22)).

Elucidating the Design Space of Text-to-Audio Models

Published:

ETTA is a text-to-audio model trained on publicly available audio datasets with synthetic captions. ETTA significantly outperforms open-sourced baseline models and is comparable to models trained on proprietary data. Furthermore, ETTA has an improved ability to generate creative audio using imaginative text prompts.

NVLM: Open Frontier-Class Multimodal LLMs

Published:

We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.

RADMMM: Multilingual Multiaccented Multispeaker TTS with RADTTS

Published:

We present an multi-lingual multi-accented multi-speaker (MMM) speech synthesis system extending on our previous work with RADTTS, RADTTS++ and Alignment Learning Framework. Our method doesn’t rely on data with speaker(s) speaking multiple languages and allows generating speech in a desired language seen by the model with the proper accent while retaining the characteristics of an individual voice.

BigVGAN: A Universal Neural Vocoder with Large-Scale Training

Published:

we present BigVGAN, a universal neural vocoder. It’s trained only on speech data but shows extraordinary zero-shot generalization ability for non-speech vocalizations (laughter, applaud), singing voices, music, instrumental audio that are even recorded in varied noisy environment!

Speech Denoising in the Waveform Domain with Self-Attention

Published:

We present CleanUNet, a speech denoising model on the raw waveform. It is based on an encoder-decoder architecture combined with several self-attention blocks to refine its bottleneck representations, which is crucial to obtain good results. It outperforms the state-of-the-art models in terms of denoised speech quality from various objective and subjective evaluation metrics.

One TTS Alignment to Rule Them All

Published:

We present an unsupervised alignment learning framework that learns speech-text alignments online in text to speech models. We showcase this alignment learning framework can be applied to any TTS model removing the dependency of TTS systems on external aligners. It also enhances the speech quality as evaluated by human evaluators.

View Generalization for Single Image Textured 3D Models

Published:

Recommended citation: Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, Bryan Catanzaro, View Generalization for Single Image Textured 3D Models, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR) 2021.

MegatronLM’s Supercharged V1.0

Published:

We release version 1.0 of Megatron which makes the training of large NLP models even faster and sustains 62.4 teraFLOPs in the end-to-end training that is 48% of the theoretical peak FLOPS for a single GPU in a DGX2-H server.

Unsupervised Video Interpolation Using Cycle Consistency

Published:

We propose unsupervised techniques to synthesize high frame rate videos directly from low frame rate videos using cycle consistency. We also introduce a pseudo-supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model. The pseudo-supervised loss term, used together with cycle consistency, can effectively adapt a pre-trained model to a new target domain. We show results that significantly reduce the domain gap problem in video frame interpolation.

Recommended citation: Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro, "Unsupervised Video Interpolation Using Cycle Consistency". In ICCV 2019. https://arxiv.org/abs/1906.05928

Improving Semantic Segmentation via Video Propagation and Label Relaxation

Published:

This paper shows how to scale up training sets for semantic segmentation by using video prediction-based data synthesis method. Our proposed joint propagation strategy and boundary relaxation technique can alleviate the label noise in the synthesized samples and lead to state-of-the-art performance on three benchmark datasets Cityscapes, CamVid and KITTI.

Recommended citation: Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao and Bryan Catanzaro, Improving Semantic Segmentation via Video Propagation and Label Relaxation, arXiv:1812.01593, 2018. https://arxiv.org/abs/1812.01593

SDCNet: Video Prediction Using Spatially Displaced Convolution

Published:

SDCNet is a 3D convolutional neural network proposed for frame prediction. The model takes as input a sequence of past frames and their inter-frame optical flows and generates a per-pixel kernel and motion vector. A future frame is then synthesised by sampling past frames guided by the motion vectors and weighted by the learned kernels.

Recommended citation: Fitsum A. Reda, Guilin Liu, Kevin J. Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, Bryan Catanzaro, SDCNet: Video Prediction Using Spatially Displaced Convolution. ECCV 2018. https://arxiv.org/abs/1811.00684

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Published:

This paper shows how to do large scale distributed, large batch, mixed precision training of language models with investigations into the successes and limitations of large batch training on publicly available language datasets.

Recommended citation: Raul Puri, Robert Kirby, Nikolai Yakovenko, Bryan Catanzaro, Large Scale Language Modeling: Converging on 40GB of Text in Four Hours. arXiv. 2018. https://arxiv.org/abs/1808.01371

Malware Detection by Eating a Whole EXE

Published:

This paper shows how to do whole binary classification for malware detection with a convolutional neural network. Done in collaboration with researchers at the University of Maryland.

Recommended citation: Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, Charles Nicholas, Malware Detection by Eating a Whole EXE. arXiv. 2017. http://arxiv.org/abs/1710.09435