RADMMM: Multilingual Multiaccented Multispeaker TTS with RADTTS


We present an multi-lingual multi-accented multi-speaker (MMM) speech synthesis system extending on our previous work with RADTTS, RADTTS++ and Alignment Learning Framework. Our method doesn’t rely on data with speaker(s) speaking multiple languages and allows generating speech in a desired language seen by the model with the proper accent while retaining the characteristics of an individual voice.

BigVGAN: A Universal Neural Vocoder with Large-Scale Training


we present BigVGAN, a universal neural vocoder. It’s trained only on speech data but shows extraordinary zero-shot generalization ability for non-speech vocalizations (laughter, applaud), singing voices, music, instrumental audio that are even recorded in varied noisy environment!

Speech Denoising in the Waveform Domain with Self-Attention


We present CleanUNet, a speech denoising model on the raw waveform. It is based on an encoder-decoder architecture combined with several self-attention blocks to refine its bottleneck representations, which is crucial to obtain good results. It outperforms the state-of-the-art models in terms of denoised speech quality from various objective and subjective evaluation metrics.

One TTS Alignment to Rule Them All


We present an unsupervised alignment learning framework that learns speech-text alignments online in text to speech models. We showcase this alignment learning framework can be applied to any TTS model removing the dependency of TTS systems on external aligners. It also enhances the speech quality as evaluated by human evaluators.

View Generalization for Single Image Textured 3D Models


Recommended citation: Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, Bryan Catanzaro, View Generalization for Single Image Textured 3D Models, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR) 2021.

MegatronLM’s Supercharged V1.0


We release version 1.0 of Megatron which makes the training of large NLP models even faster and sustains 62.4 teraFLOPs in the end-to-end training that is 48% of the theoretical peak FLOPS for a single GPU in a DGX2-H server.

Unsupervised Video Interpolation Using Cycle Consistency


We propose unsupervised techniques to synthesize high frame rate videos directly from low frame rate videos using cycle consistency. We also introduce a pseudo-supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model. The pseudo-supervised loss term, used together with cycle consistency, can effectively adapt a pre-trained model to a new target domain. We show results that significantly reduce the domain gap problem in video frame interpolation.

Recommended citation: Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro, "Unsupervised Video Interpolation Using Cycle Consistency". In ICCV 2019.

Improving Semantic Segmentation via Video Propagation and Label Relaxation


This paper shows how to scale up training sets for semantic segmentation by using video prediction-based data synthesis method. Our proposed joint propagation strategy and boundary relaxation technique can alleviate the label noise in the synthesized samples and lead to state-of-the-art performance on three benchmark datasets Cityscapes, CamVid and KITTI.

Recommended citation: Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao and Bryan Catanzaro, Improving Semantic Segmentation via Video Propagation and Label Relaxation, arXiv:1812.01593, 2018.

SDCNet: Video Prediction Using Spatially Displaced Convolution


SDCNet is a 3D convolutional neural network proposed for frame prediction. The model takes as input a sequence of past frames and their inter-frame optical flows and generates a per-pixel kernel and motion vector. A future frame is then synthesised by sampling past frames guided by the motion vectors and weighted by the learned kernels.

Recommended citation: Fitsum A. Reda, Guilin Liu, Kevin J. Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, Bryan Catanzaro, SDCNet: Video Prediction Using Spatially Displaced Convolution. ECCV 2018.

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours


This paper shows how to do large scale distributed, large batch, mixed precision training of language models with investigations into the successes and limitations of large batch training on publicly available language datasets.

Recommended citation: Raul Puri, Robert Kirby, Nikolai Yakovenko, Bryan Catanzaro, Large Scale Language Modeling: Converging on 40GB of Text in Four Hours. arXiv. 2018.

Malware Detection by Eating a Whole EXE


This paper shows how to do whole binary classification for malware detection with a convolutional neural network. Done in collaboration with researchers at the University of Maryland.

Recommended citation: Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, Charles Nicholas, Malware Detection by Eating a Whole EXE. arXiv. 2017.