NVIDIA Research Taiwan

NVIDIA Research Taiwan

Introduction

Welcome to the homepage of the NVIDIA Research Taiwan. Founded in 2022, NVR Taiwan is the APAC research team of NVIDIA Research, with primary research focuses on Generative AI, multimodal learning and 3D vision. The team is led by Professor Frank Wang.

We invite applications for the following positions:

  • Full-time research scientist
  • Research internship

Graduate students with strong research publications and interested in doing an internship at NVR Taiwan can see this link for open positions, or contact our members for more details.

News

Feb. 2026 - 7 papers have been accepted at CVPR 2026 (Main) and 2 papers at CVPR (Findings).

Jan. 2026 - 3 papers have been accepted at ICRA 2026.

Dec. 2025 - 3 papers have been accepted at WACV 2026.

Dec. 2025 - 5 papers have been accepted at NeurIPS 2025.

Oct. 2025 - 2 papers have been accepted at ICCV 2025.

Mar. 2025 - 10 papers have been accepted at CVPR 2025.

Feb. 2025 - 7 papers have been accepted at ICLR 2025.

Oct. 2024 - 3 papers have been accepted at WACV 2025.

Sep. 2024 - 3 papers at NeurIPS 2024 and 2 papers at EMNLP 2024 (Main) have been accepted.

Aug. 2024 - 1 paper has been accepted at WACV 2025 (1st Round). 3 papers have been accepted at IEEE SLT 2024.

Members

Avatar

Yu-Chiang Frank Wang

Avatar

Min-Hung Chen

Avatar

Szu-Wei Fu

Avatar

Ryo Hachiuma

Avatar

Cheng Sun

Avatar

Jaesung Choe

Avatar

Huck Yang

Avatar

Fred Yang

Avatar

Sung-Feng Huang

Avatar

Yusuke Hirota

Avatar

ByungKwan Lee

Avatar

Chi-Pin Huang

Avatar

(Design Automation Research) Wen-Hao Liu

Avatar

(Design Automation Research) Danny Liu

Current interns

Avatar

Joe Yang

Avatar

Seokju Cho

Avatar

Minki Kang

Avatar

Chin-Yang Lin

Avatar

Yu-Ji Kim

Avatar

Rikiya Takehi

Alumni

Avatar

Sheng-Yu Huang

Avatar

Ci-Siang Lin

Avatar

Chien-Yi Wang

Avatar

Yun Da Tsai

Avatar

Shih-Yang Liu

Avatar

Hsu-kuang Chiu

Avatar

Mu-Yi Shen

Avatar

Shun-Cheng Wu

Avatar

Yen-Ting Lin

Avatar

Kris Wu

Avatar

Chunghyun Park

Avatar

Kai-Po Chang

Avatar

Miran Heo

Avatar

Yoonwoo Jeong

Avatar

Zhen Wan

Avatar

Xiaoqian Shen

Research Topics

A non-exhaustive list of research topics studied in our group include:

  • Vision & Language

  • Audio & Speech

  • Multimodal Learning

  • 3D Vision

  • Natural Language Processing

Publications

Quickly discover relevant content by filtering publications.
*

4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

CVPR 2026

Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, …
PDF Project
4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning

CVPR 2026

Vision-Language-Action (VLA) tasks require reasoning over complex visual scenes and executing adaptive actions in dynamic environments. …
PDF Project
Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning

Frequency Switching Mechanism for Parameter-Efficient Multi-Task Learning

CVPR 2026

Multi-task learning (MTL) aims to equip a single model with the ability to solve multiple tasks efficiently; however, current …
Frequency Switching Mechanism for Parameter-Efficient Multi-Task Learning

Interpretable Debiasing of Vision-Language Models for Social Fairness

CVPR 2026

The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead …
PDF
Interpretable Debiasing of Vision-Language Models for Social Fairness

Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

CVPR 2026

Large-scale vision-language models (VLMs) have recently achieved remarkable multimodal understanding, but their massive size makes them …
PDF
Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

CVPR 2026

We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene …
PDF Project
OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding

CVPR 2026

Video Large Language Models (VideoLLMs) have shown remarkable progress in video understanding. However, these models still struggle to …
PDF
SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding

Autoregressive Universal Video Segmentation Model

CVPR 2026 (Findings)

Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. …
PDF
Autoregressive Universal Video Segmentation Model

Recursive Think-Answer Process for LLMs and VLMs

CVPR 2026 (Findings)

Think-Answer reasoners such as DeepSeek-R1 have made notable progress by leveraging interpretable internal reasoning. However, despite …
PDF Project
Recursive Think-Answer Process for LLMs and VLMs

V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts

ICRA 2026, ICCVW 2025

Current state-of-the-art autonomous vehicles could face safety-critical situations when their local sensors are occluded by large …
PDF Code Dataset Project
V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models

ICRA 2026, CVPRW 2025 (Oral)

Current autonomous driving vehicles rely mainly on their individual sensors to understand surrounding scenes and plan for future …
PDF Code Dataset Project
V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models

RealTraj: Towards Real-World Pedestrian Trajectory Forecasting

ICRA 2026

This paper jointly addresses three key limitations in conventional pedestrian trajectory forecasting: pedestrian perception errors, …
PDF Project
RealTraj: Towards Real-World Pedestrian Trajectory Forecasting

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

ICLR 2026

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. …
PDF
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

RefineBench: Evaluating Refinement Capability of Language Models via Checklists

ICLR 2026, NeurIPSW 2025

We revisit the question of whether or not language models (LMs) can refine their responses. Extending beyond prior works, we explore …
PDF Code Dataset Project Video
RefineBench: Evaluating Refinement Capability of Language Models via Checklists

Test-Time Alignment for Large Language Models via Textual Model Predictive Control

ICLR 2026

Aligning Large Language Models (LLMs) with human preferences through finetuning is resource-intensive, motivating lightweight …
PDF
Test-Time Alignment for Large Language Models via Textual Model Predictive Control

EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

ICLRW 2026

While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language …
PDF Code Source Document
EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in

ICLRW 2026

Grounded video question answering (GVQA) aims to localize relevant temporal segments in videos and generate accurate answers to a given …
PDF
Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in

Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment

WACV 2026

Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input …
PDF Project
Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment

TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors

WACV 2026

Dense video captioning aims to interpret and describe all temporally localized events throughout an input video. Recent …
PDF
TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors

VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models

WACV 2026 (Oral)

Video anomaly understanding (VAU) aims to provide detailed interpretation and semantic comprehension of anomalous events within videos, …
PDF Project
VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models

TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control

NeurIPSW 2025

Current controllable diffusion models typically rely on fixed architectures that modify intermediate activations to inject guidance …
PDF Project
TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control

BlurDM: A Blur Diffusion Model for Image Deblurring

NeurIPS 2025

Diffusion models show promise for dynamic scene deblurring; however, existing studies often fail to leverage the intrinsic nature of …
PDF Project
BlurDM: A Blur Diffusion Model for Image Deblurring

ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning

NeurIPS 2025

Vision-language-action (VLA) reasoning tasks require agents to interpret multimodal instructions, perform long-horizon planning, and …
PDF Project
ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning

EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction

NeurIPS 2025

Open-source foundation models have seen rapid adoption and development, enabling powerful general-purpose capabilities across diverse …
PDF Code Project
EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction

Unified Reinforcement and Imitation Learning for Vision-Language Models

NeurIPS 2025

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for …
PDF Project
Unified Reinforcement and Imitation Learning for Vision-Language Models

Towards Predicting Any Human Trajectory In Context

NeurIPS 2025

Predicting accurate future trajectories of pedestrians is essential for autonomous systems but remains a challenging task due to the …
PDF Project
Towards Predicting Any Human Trajectory In Context

Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation

ICCV 2025

Gender bias in vision-language foundation models (VLMs) raises concerns about their safe deployment and is typically evaluated using …
PDF
Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation

LongSplat: Robust Unposed 3D Gaussian Splatting for Casual Long Videos

ICCV 2025

LongSplat addresses critical challenges in novel view synthesis (NVS) from casually captured long videos characterized by irregular …
PDF Code Project
LongSplat: Robust Unposed 3D Gaussian Splatting for Casual Long Videos

VoiceNoNG: High-Quality Speech Editing Model without Hallucinations

Interspeech 2025

Voicebox and VoiceCraft are the current most representative models for non-autoregressive and autoregressive speech editing, …
PDF
VoiceNoNG: High-Quality Speech Editing Model without Hallucinations

VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

CVPR 2025

The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has …
PDF Project
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models

CVPR 2025

Customized text-to-video generation aims to produce high-quality videos that incorporate user-specified subject identities or motion …
PDF
VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models

3D Gaussian Inpainting with Depth-Guided Cross-View Consistency

CVPR 2025

When performing 3D inpainting using novel-view rendering methods like Neural Radiance Field (NeRF) or 3D Gaussian Splatting (3DGS), how …
PDF
3D Gaussian Inpainting with Depth-Guided Cross-View Consistency

AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting

CVPR 2025

Three-dimensional scene inpainting is crucial for applications from virtual reality to architectural visualization, yet existing …
PDF Project
AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360° Unbounded Scene Inpainting

Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

CVPR 2025

We introduce Dr. Splat, a novel approach for openvocabulary 3D scene understanding leveraging 3D Gaussian Splatting. Unlike existing …
PDF Project
Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

FrugalNeRF: Fast Convergence for Extreme Few-shot Novel View Synthesis without Learned Priors

CVPR 2025

Neural Radiance Fields (NeRF) face significant challenges in extreme few-shot scenarios, primarily due to overfitting and long training …
PDF Project
FrugalNeRF: Fast Convergence for Extreme Few-shot Novel View Synthesis without Learned Priors

Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation

CVPR 2025

We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method …
PDF Project
Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation

Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks

CVPR 2025

We present Omni-RGPT, a multimodal large language model designed to facilitate region-level comprehension for both images and videos. …
PDF Project
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks

Segment Anything, Even Occluded

CVPR 2025

Amodal instance segmentation, which aims to detect and segment both visible and invisible parts of objects in images, plays a crucial …
Segment Anything, Even Occluded

Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering

CVPR 2025

We propose an efficient radiance field rendering algorithm that incorporates a rasterization process on adaptive sparse voxels without …
PDF Code Project
Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Rendering

Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

ICLR 2025

An ideal multimodal agent should be aware of the quality of its input modalities. Recent advances have enabled large language models …
PDF
Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL

ICLR 2025

Bayesian optimization (BO) offers an efficient pipeline for optimizing black-box functions with the help of a Gaussian process prior …
PDF
BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

ICLR 2025

Multimodal foundation models, such as Gemini and GPT-4, have revolutionized human-machine interactions by seamlessly integrating …
PDF
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

Hymba: A Hybrid-head Architecture for Small Language Models

ICLR 2025

The transformative capabilities of language models (LMs) have intensified the demand for their deployment on everyday devices, …
PDF Code Source Document
Hymba: A Hybrid-head Architecture for Small Language Models

SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

ICLR 2025

Large-scale vision-language models, such as CLIP, are known to contain societal bias regarding protected attributes (e.g., gender, …
PDF Project
SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

Towards Neural Scaling Laws for Time Series Foundation Models

ICLR 2025

Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely …
PDF
Towards Neural Scaling Laws for Time Series Foundation Models

UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation

ICLR 2025

Pre-training and representation learning have been playing an increasingly important role in modern speech processing. Nevertheless, …
PDF Project
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation

CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

WACV 2025

In the task of reference-based image inpainting, an additional reference image is provided to restore a damaged target image to its …
PDF Code Project
CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

Spatio-Temporal Context Prompting for Zero-Shot Action Detection

WACV 2025

Spatio-temporal action detection encompasses the tasks of localizing and classifying individual actions within a video. Recent works …
PDF Code Project
Spatio-Temporal Context Prompting for Zero-Shot Action Detection

Diffusion-Reward Adversarial Imitation Learning

NeurIPS 2024

Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. …
PDF Code Project
Diffusion-Reward Adversarial Imitation Learning

RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

NeurIPS 2024

State-of-the-art (SOTA) semi-supervised learning techniques, such as FixMatch and it’s variants, have demonstrated impressive …
PDF
RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

NeurIPS 2024

We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the …
PDF
Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

EMNLP 2024

Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue …
PDF
Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

EMNLP 2024

Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. This generative approach to …
PDF Code
From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation

WACV 2025

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation models using image data with only image-level supervision. …
PDF Code Project
Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation

An Investigation of Incorporating Mamba for Speech Enhancement

SLT 2024

This work aims to study a scalable state-space model (SSM), Mamba, for the speech enhancement (SE) task. We exploit a Mamba-based …
PDF
An Investigation of Incorporating Mamba for Speech Enhancement

Detecting the Undetectable: Assessing the Efficacy of Current Spoof Detection Methods Against Seamless Speech Edits

SLT 2024

Neural speech editing advancements have raised concerns about their misuse in spoofing attacks. Traditional partially edited speech …
Detecting the Undetectable: Assessing the Efficacy of Current Spoof Detection Methods Against Seamless Speech Edits

The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

SLT 2024

We present the third edition of the VoiceMOS Challenge, a scientific initiative designed to advance research into automatic prediction …
PDF
The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

ECCV 2024

Concept erasure in text-to-image diffusion models aims to disable pre-trained diffusion models from generating images related to a …
PDF
Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

ECCV 2024

We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models …
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

ECCV 2024

Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, when …
PDF
Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition

ECCV 2024

We address a novel cross-domain few-shot learning task (CD-FSL) with multimodal input and unlabeled target data for egocentric action …
PDF Project
Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition

Desta: Enhancing speech language models through descriptive speech-text alignment

Interspeech 2024

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language …
PDF
Desta: Enhancing speech language models through descriptive speech-text alignment

GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

ACL 2024

Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by …
PDF Code
GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

DoRA: Weight-Decomposed Low-Rank Adaptation

ICML 2024 (Oral)

Among the widely used parameter-efficient finetuning (PEFT) methods, LoRA and its variants have gained considerable popularity because …
PDF Code Project Source Document
DoRA: Weight-Decomposed Low-Rank Adaptation

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

CVPRW 2024

Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. …
PDF Project
GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

CVPR 2024

Utilizing multi-view inputs to synthesize novel-view images, Neural Radiance Fields (NeRF) have emerged as a popular research topic in …
PDF
GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes

CVPR 2024

Recent advancements in post-hoc and inherently interpretable methods have markedly enhanced the explanations of black box classifier …
MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes

Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

CVPR 2024

State-of-the-art single-view 360-degree room layout reconstruction methods formulate the problem as a high-level 1D (per-column) …
PDF
Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

ICRA 2024

Current state-of-the-art autonomous driving vehicles mainly rely on each individual sensor system to perform perception tasks. Such a …
PDF Code Project
Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

ICLR 2024

Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) …
PDF
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

ICLR 2024

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), …
PDF
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

ICLR 2024

Natural Language Explanation (NLE) in vision and language tasks aims to provide human-understandable explanations for the associated …
PDF
RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech

ICLR 2024

Speech quality estimation has recently undergone a paradigm shift from human- hearing expert designs to machine-learning models. …
PDF
Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

NeurIPS 2023

Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly …
PDF Code
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

NeurIPS 2023

Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its …
PDF Code
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

EMNLP 2023

We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our …
PDF Code
Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

ICCV 2023

We present a Multimodal Interlaced Transformer (MIT) that jointly considers 2D and 3D data for weakly supervised point cloud …
PDF Code Project
2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation

ICCV 2023

Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without …
PDF
Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation

Neural-PBIR Reconstruction of Shape, Material, and Illumination

ICCV 2023

Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination …
PDF Project
Neural-PBIR Reconstruction of Shape, Material, and Illumination

Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction

ICCV 2023

We propose an algorithm, 4DRegSDF, for the spacetime surface regularization to improve the fidelity of neural rendering and …
Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction

Frequency-Aware Self-Supervised Long-Tailed Learning

ICCVW 2023

Data collected from the real world typically exhibit long-tailed distributions, where frequent classes contain abundant data while rare …
PDF
Frequency-Aware Self-Supervised Long-Tailed Learning

Bias-Eliminating Augmentation Learning for Debiased Federated Learning

CVPR 2023

Learning models trained on biased datasets tend to observe correlations between categorical and undesirable features, which result in …
PDF
Bias-Eliminating Augmentation Learning for Debiased Federated Learning

QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

CVPRW 2023

This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the …
PDF Code
QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

AAAI 2023

Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with …
PDF Code
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

Target-Free Text-guided Image Manipulation

AAAI 2023

We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on …
PDF Project
Target-Free Text-guided Image Manipulation

Paraphrasing Is All You Need for Novel Object Captioning

NeurIPS 2022

Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. …
PDF
Paraphrasing Is All You Need for Novel Object Captioning

SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion

NeurIPS 2022

Point cloud completion is an active research topic for 3D vision and has been widely studied in recent years. Instead of directly …
PDF
SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion

Contact

Our lab is mainly located in Taipei City (10-15 minutes away from Taipei Songshan Airport by MRT). Motivated candidates can contact our members to apply for internships and research positions.

Unauthorized visitors are not permitted in the Taipei office.

  • No. 8, Jihu Rd, Nangang District, Taipei City, 114
  • Monday-Friday 9:00 to 18:00