Unified Reinforcement and Imitation Learning for Vision-Language Models

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning.

VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

The recent surge in high-quality visual instruction tuning samples from closed-source vision-language models (VLMs) such as GPT-4V has accelerated the release of open-source VLMs across various model sizes. However, scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots. To address this, we propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which prioritizes efficiency without compromising accuracy.

Effects of Frame Rates up to 500 Hz on First Person Shooter Game Players

Computer games – and computer game players – often drive technology improvements, with graphics cards and monitors pushing the limits of display technologies. High frame rates, in particular, promise to provide lower latencies and smoother game visuals to gamers, especially important for competitive first-person shooter (FPS) game players. What is not well-known is to what extent gamers benefit from ultra-high frame rates in terms of player performance and quality of experience. This paper studies the effects of frame rates – especially high frame rates – on FPS game players.

Addressable Memory for Video World Models

We study visual persistence in interactive video world models. These models rely on a Key-Value (KV) cache as a growing visual memory to carry forward previously generated frames. However, we find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal Rotary Positional Embeddings (RoPE) offsets then fall outside the range seen during training and the model struggles to retrieve the relevant visual information through attention.

Motion Attribution for Video Generation

Despite the rapid progress of video generation models, the role of data in influencing motion is poorly understood. We present Motive (MOTIon attribution for Video gEneration), a motion-centric, gradient-based data attribution framework that scales to modern, large, high-quality video datasets and models. We use this to study which fine-tuning clips improve or degrade temporal dynamics. Motive isolates temporal dynamics from static appearance via motion-weighted loss masks, yielding efficient and scalable motion-specific influence computation.

Variance Reduction for Expectations with Diffusion Teachers

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo (MC) expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work (rendering, simulation, encoding).

Jonathan Lorraine

Jonathan Lorraine is a Senior Research Scientist at NVIDIA, in the Learning and Perception Research (LPR) lab in Toronto. His current work is on agentic AI for science and engineering: agents that propose, run and verify their own experiments, and the measurement discipline that makes their results trustworthy.

ROSA: A Robotics Foundation Model Serving System for Robot Factories

Robotics foundation models (RFMs) are making general-purpose robots increasingly practical for factory deployments. While RFM serving systems are central to this vision, existing systems are largely shaped by a single-robot, single-model assumption: inference is treated as an edge-computing problem handled by an on-robot or dedicated nearby GPU, and the serving objective is to minimize the latency of a single action model. In this paper, we propose ROSA, an RFM serving system for robot factories designed around three key principles.

Kaiyang Zhao

Kaiyang's research focuses on operating systems and architecture with an emphasis on efficiency, performance and scalability in data centers.

Kaiyang holds a PhD degree from Carnegie Mellon University and a BS degree from Purdue University.

For more information, visit my home page.