Publications

Spatio-Temporal Context Prompting for Zero-Shot Action Detection

WACV 2025

Spatio-temporal action detection encompasses the tasks of localizing and classifying individual actions within a video. Recent works …
PDF Project
Spatio-Temporal Context Prompting for Zero-Shot Action Detection

CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

WACV 2025

In the task of reference-based image inpainting, an additional reference image is provided to restore a damaged target image to its …
CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

NeurIPS 2024

We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the …
PDF
Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

NeurIPS 2024

State-of-the-art (SOTA) semi-supervised learning techniques, such as FixMatch and it’s variants, have demonstrated impressive …
PDF
RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

Diffusion-Reward Adversarial Imitation Learning

NeurIPS 2024

Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. …
PDF Code Project
Diffusion-Reward Adversarial Imitation Learning

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

EMNLP 2024

Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. This generative approach to …
PDF Code
From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

EMNLP 2024

Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue …
PDF
Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation

WACV 2025

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation models using image data with only image-level supervision. …
PDF
Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation

The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

SLT 2024

We present the third edition of the VoiceMOS Challenge, a scientific initiative designed to advance research into automatic prediction …
PDF
The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction

Detecting the Undetectable: Assessing the Efficacy of Current Spoof Detection Methods Against Seamless Speech Edits

SLT 2024

Neural speech editing advancements have raised concerns about their misuse in spoofing attacks. Traditional partially edited speech …
Detecting the Undetectable: Assessing the Efficacy of Current Spoof Detection Methods Against Seamless Speech Edits

An Investigation of Incorporating Mamba for Speech Enhancement

SLT 2024

This work aims to study a scalable state-space model (SSM), Mamba, for the speech enhancement (SE) task. We exploit a Mamba-based …
PDF
An Investigation of Incorporating Mamba for Speech Enhancement

Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

ECCV 2024

Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, when …
PDF
Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

ECCV 2024

We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models …
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

ECCV 2024

Concept erasure in text-to-image diffusion models aims to disable pre-trained diffusion models from generating images related to a …
PDF
Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition

ECCV 2024

We address a novel cross-domain few-shot learning task (CD-FSL) with multimodal input and unlabeled target data for egocentric action …
PDF Project
Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition

Desta: Enhancing speech language models through descriptive speech-text alignment

Interspeech 2024

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language …
PDF
Desta: Enhancing speech language models through descriptive speech-text alignment

GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

ACL 2024

Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by …
PDF Code
GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

DoRA: Weight-Decomposed Low-Rank Adaptation

ICML 2024

Among the widely used parameter-efficient finetuning (PEFT) methods, LoRA and its variants have gained considerable popularity because …
PDF Code Project Source Document
DoRA: Weight-Decomposed Low-Rank Adaptation

GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

CVPRW 2024

Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. …
PDF Project
GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

CVPR 2024

State-of-the-art single-view 360-degree room layout reconstruction methods formulate the problem as a high-level 1D (per-column) …
PDF
Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes

CVPR 2024

Recent advancements in post-hoc and inherently interpretable methods have markedly enhanced the explanations of black box classifier …
MCPNet: An Interpretable Classifier via Multi-Level Concept Prototypes

GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

CVPR 2024

Utilizing multi-view inputs to synthesize novel-view images, Neural Radiance Fields (NeRF) have emerged as a popular research topic in …
PDF
GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

ICRA 2024

Current state-of-the-art autonomous driving vehicles mainly rely on each individual sensor system to perform perception tasks. Such a …
PDF Code Project
Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter

Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech

ICLR 2024

Speech quality estimation has recently undergone a paradigm shift from human- hearing expert designs to machine-learning models. …
PDF
Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech

RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

ICLR 2024

Natural Language Explanation (NLE) in vision and language tasks aims to provide human-understandable explanations for the associated …
PDF
RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

ICLR 2024

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), …
PDF
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

ICLR 2024

Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) …
PDF
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

EMNLP 2023

We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our …
PDF Code
Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

NeurIPS 2023

Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its …
PDF Code
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

NeurIPS 2023

Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly …
PDF Code
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation

ICCV 2023

Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without …
PDF
Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation

2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

ICCV 2023

We present a Multimodal Interlaced Transformer (MIT) that jointly considers 2D and 3D data for weakly supervised point cloud …
PDF Code Project
2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction

ICCV 2023

We propose an algorithm, 4DRegSDF, for the spacetime surface regularization to improve the fidelity of neural rendering and …
Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction

Neural-PBIR Reconstruction of Shape, Material, and Illumination

ICCV 2023

Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination …
PDF Project
Neural-PBIR Reconstruction of Shape, Material, and Illumination

Frequency-Aware Self-Supervised Long-Tailed Learning

ICCVW 2023

Data collected from the real world typically exhibit long-tailed distributions, where frequent classes contain abundant data while rare …
PDF
Frequency-Aware Self-Supervised Long-Tailed Learning

QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

CVPRW 2023

This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the …
PDF Code
QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

Bias-Eliminating Augmentation Learning for Debiased Federated Learning

CVPR 2023

Learning models trained on biased datasets tend to observe correlations between categorical and undesirable features, which result in …
PDF
Bias-Eliminating Augmentation Learning for Debiased Federated Learning

Target-Free Text-guided Image Manipulation

AAAI 2023

We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on …
PDF Project
Target-Free Text-guided Image Manipulation

Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

AAAI 2023

Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with …
PDF Code
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion

NeurIPS 2022

Point cloud completion is an active research topic for 3D vision and has been widely studied in recent years. Instead of directly …
PDF
SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion

Paraphrasing Is All You Need for Novel Object Captioning

NeurIPS 2022

Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. …
PDF
Paraphrasing Is All You Need for Novel Object Captioning