Learning to Track Instances without Video Annotations

Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches. To resolve these challenges, we introduce a novel semi-supervised framework by learning instance tracking networks with only a labeled image dataset and unlabeled video sequences. With an instance contrastive objective, we learn an embedding to discriminate each instance from the others.

Weakly-Supervised Physically Unconstrained Gaze Estimation

A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios. In contrast, videos of human interactions in unconstrained environments are abundantly available and can be much more easily annotated with frame-level activity labels. In this work, we tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions.

Contrastive Syn-to-Real Generalization

Training on synthetic data can be beneficial for label or data-scarce scenarios. However, synthetically trained models often suffer from poor generalization in real domains due to domain gaps. In this work, we make a key observation that the diversity of the learned feature embeddings plays an important role in the generalization performance.

ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning

Isekai3D SDG: A Platform for Diverse Synthetic Gameplay Visual Data

We present Isekai3D SDG (Synthetic Data Generation), a platform for generating diverse 3D gameplay data for world models, computer vision, neural graphics, and gameplay agents. The platform combines reusable environments and characters with modular Unreal Engine plugins. A two-stage workflow records world simulation and then replays it to capture complementary views and rendering passes from the same underlying gameplay. Configurable cameras, resolutions, and frame rates support paired observations with frame-aligned camera, character, and action metadata.

DemoStream: Batched Task-and-Motion-Plan Optimization for Automated Learning from Demonstration

cuArms: GPU-Accelerated Conflict-Based Planning for Multi-Arm Manipulation

TAMPAgent: Agentic Belief-Space Planning for Mobile Manipulation

Imitation Learning of Avoidance Modes for Efficient Multimodal Model Predictive Control

Multimodal model predictive control (MM-MPC) avoids local minima among moving obstacles by solving one optimal control problem (OCP) per combination of avoidance sides, so its computational cost grows exponentially with the number of decision-critical obstacles. We propose SideNet-MPC, which replaces this enumeration with a mode decision learned by imitating MM-MPC as an expert. A graph neural network is trained to predict the expert's minimum-cost mode, one avoidance side per obstacle.

EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud Delegation

Complex and long-horizon agent tasks often rely on frontier models deployed in the cloud to achieve strong performance. However, using these models for every step of a long-horizon trajectory incurs substantial inference cost, even though not all steps may require the same level of model capability. In this paper, we find that allocating the LLM to only a small subset of key steps can achieve performance comparable to full LLM execution while significantly reducing inference cost.

Subscribe to