Humans are extraordinarily skilled at reading not just acoustic cues but also facial expressions, body language, and proxemics — abilities honed over a lifetime of social interaction. The most immersive interface with a machine, therefore, is one that is indistinguishable from interacting with another person. We are building toward this vision with immersive social agents, embodied as 3D Codec Avatars, that consume multimodal user input — text, speech, body motion, facial expressions, and gaze — and engage in full-duplex dyadic conversation with spatial understanding of 3D motion and gesture. Our north star is the embodied Turing test: agents that move, speak, and behave so naturally that human-machine interaction becomes indistinguishable from human-human interaction. In this talk, I will present joint generative modeling approaches for producing all relevant output modalities in concert, as well as data strategies for scaling effectively beyond dedicated 3D capture sessions.

Recent advances in simulation, reinforcement learning, and generative modeling have made it possible to bring expressive, dynamic motions to physical robots. However, as systems grow in complexity, monolithic architectures often struggle with sample inefficiency, lack of interpretability, and the immense difficulty of generalizing across diverse tasks. In this talk, I present modular building blocks that leverage human motion data to make robots move in lifelike ways.

This approach centers on three fundamental blocks: Understanding (perceiving intent and environment), Generation (synthesizing feasible motion plans), and Control (grounding motion in real-world physics). I will detail our latest research in motion tracking and generation, and how tailored learning components scale independently while remaining contextually aware of the requirements and constraints of adjacent modules. Finally, I outline how the interfaces between these blocks provide a robust pathway for world-understanding models to translate high-level intent into grounded action, highlighting why the interface between these components is critical for future robotic intelligence.

Language-based AI is now ubiquitous, and user expectations for intelligent machines are scaling along with it: we expect machines to understand us, predict our needs and wants, do what we enjoy and prefer, and adapt as we change our moods and minds, learn, grow, and age. Physical AI, in the form of robotics, is the next major AI challenge, and it is not ready to leap into our daily lives yet. While massive investment is focused on functional behavior of humanoid robots (perceiving the world, moving around, and manipulating objects), human-robot interaction (HRI) is relegated to an afterthought. It is assumed that once a robot can move around and do things, it will be useful and wanted, yet over 25 years of research in HRI tells us otherwise. While the needs for human-centered services continue to grow, research and development is minimal. This talk will discuss how bringing together robotics, AI, and machine learning for long-term user modeling, real-time multimodal behavioral signal processing, and affective computing is enabling machines to understand, interact, and adapt to users' specific and ever-changing needs. We will overview methods and challenges of sparse and noisy heterogeneous, multi-modal, personal interaction data and of creating expressive agent and robot behavior toward understanding, coaching, motivating, and supporting a wide variety of user populations across the age span (infants, children, adults, elderly), ability span (typically developing, autism, anxiety, stroke, dementia), contexts (schools, therapy centers, homes), and deployment durations (from weeks to 6 months) through socially assistive robotics. We will discuss the challenges of understanding what we humans want from interactions with machines vs. what we need vs. what we are getting, and how those distinctions are shaping the future of not just AI and ML but society at large.