Published:
We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.
Published:
P-Flow is a fast and data-efficient zero-shot text-to-speech model that uses speech prompting mechanism for speaker adaptation.
Published:
Recommended citation: Aysegul Dundar, Jun Gao, Andrew Tao, Bryan Catanzaro, Progressive Learning of 3D Reconstruction Network from 2D GAN Data.
Published:
We present an multi-lingual multi-accented multi-speaker (MMM) speech synthesis system extending on our previous work with RADTTS, RADTTS++ and Alignment Learning Framework. Our method doesn’t rely on data with speaker(s) speaking multiple languages and allows generating speech in a desired language seen by the model with the proper accent while retaining the characteristics of an individual voice.
Published:
we present BigVGAN, a universal neural vocoder. It’s trained only on speech data but shows extraordinary zero-shot generalization ability for non-speech vocalizations (laughter, applaud), singing voices, music, instrumental audio that are even recorded in varied noisy environment!
Published:
Recommended citation: Aysegul Dundar, Jun Gao, Andrew Tao, Bryan Catanzaro, Fine Detailed Texture Learning for 3D Meshes with Generative Models, arXiv:2203.09362, 2022. https://arxiv.org/abs/2203.09362
Published:
We present CleanUNet, a speech denoising model on the raw waveform. It is based on an encoder-decoder architecture combined with several self-attention blocks to refine its bottleneck representations, which is crucial to obtain good results. It outperforms the state-of-the-art models in terms of denoised speech quality from various objective and subjective evaluation metrics.
Published:
We present an unsupervised alignment learning framework that learns speech-text alignments online in text to speech models. We showcase this alignment learning framework can be applied to any TTS model removing the dependency of TTS systems on external aligners. It also enhances the speech quality as evaluated by human evaluators.
Published:
RAD-TTS is a parallel flow-based generative network for text-to-speech synthesis which does not rely on external aligners to learn speech-text alignments and supports diversity in generated speech by modeling speech rhythm as a separate generative distribution.
Published:
Long-Short Transformer is an efficient self-attention mechanism for modeling long sequences with linear complexity for both language and vision tasks
Published:
Recommended citation: Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, Bryan Catanzaro, View Generalization for Single Image Textured 3D Models, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR) 2021.
Published:
We release version 1.0 of Megatron which makes the training of large NLP models even faster and sustains 62.4 teraFLOPs in the end-to-end training that is 48% of the theoretical peak FLOPS for a single GPU in a DGX2-H server.
Published:
Flowtron is an autoregressive flow-based generative network for text-to-speech synthesis with direct control over speech variation and style transfer
Published:
Mellotron is a multispeaker voice synthesis model that can make a voice emote and sing without emotive or singing training data
Published:
We propose unsupervised techniques to synthesize high frame rate videos directly from low frame rate videos using cycle consistency. We also introduce a pseudo-supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model. The pseudo-supervised loss term, used together with cycle consistency, can effectively adapt a pre-trained model to a new target domain. We show results that significantly reduce the domain gap problem in video frame interpolation.
Recommended citation: Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro, "Unsupervised Video Interpolation Using Cycle Consistency". In ICCV 2019. https://arxiv.org/abs/1906.05928
Published:
We train an 8.3 billion parameter transformer language model with 8-way model parallelism and 64-way data parallelism on 512 GPUs, making it the largest transformer based language model ever trained at 24x the size of BERT and 5.6x the size of GPT-2
Published:
Recommended citation: Guilin Liu, Kevin J. Shih, Ting-Chun Wang, Fitsum A. Reda, Karan Sapra, Zhiding Yu, Andrew Tao, Bryan Catanzaro, Partial Convolution based Padding, arXiv:1811.11718, 2018. https://arxiv.org/abs/1811.11718
Published:
Recommended citation: Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro, Image Inpainting for Irregular Holes Using Partial Convolutions, Proceedings of the European Conference on Computer Vision (ECCV) 2018. https://arxiv.org/abs/1804.07723
Published:
This paper shows how to scale up training sets for semantic segmentation by using video prediction-based data synthesis method. Our proposed joint propagation strategy and boundary relaxation technique can alleviate the label noise in the synthesized samples and lead to state-of-the-art performance on three benchmark datasets Cityscapes, CamVid and KITTI.
Recommended citation: Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao and Bryan Catanzaro, Improving Semantic Segmentation via Video Propagation and Label Relaxation, arXiv:1812.01593, 2018. https://arxiv.org/abs/1812.01593
Published:
WaveGlow is an invertible neural network that can generate high quality speech efficiently from mel-spectrograms.
Published:
SDCNet is a 3D convolutional neural network proposed for frame prediction. The model takes as input a sequence of past frames and their inter-frame optical flows and generates a per-pixel kernel and motion vector. A future frame is then synthesised by sampling past frames guided by the motion vectors and weighted by the learned kernels.
Recommended citation: Fitsum A. Reda, Guilin Liu, Kevin J. Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, Bryan Catanzaro, SDCNet: Video Prediction Using Spatially Displaced Convolution. ECCV 2018. https://arxiv.org/abs/1811.00684
Published:
This paper shows how to do large scale distributed, large batch, mixed precision training of language models with investigations into the successes and limitations of large batch training on publicly available language datasets.
Recommended citation: Raul Puri, Robert Kirby, Nikolai Yakovenko, Bryan Catanzaro, Large Scale Language Modeling: Converging on 40GB of Text in Four Hours. arXiv. 2018. https://arxiv.org/abs/1808.01371
Published:
This paper shows how to do whole binary classification for malware detection with a convolutional neural network. Done in collaboration with researchers at the University of Maryland.
Recommended citation: Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, Charles Nicholas, Malware Detection by Eating a Whole EXE. arXiv. 2017. http://arxiv.org/abs/1710.09435