Learning and Perception Research

MAWAM: A Multi-Agent World-Action Model for Coordinated Manipulation

Xianzhe Fan1,2 Chi-Pin Huang1 Fu-En Yang1 Jing Wang1 Yecheng Wu1,3 Yu-Cheng Chou1 Johan Bjorck1 Linxi “Jim” Fan1 Jan Kautz1 Yu-Chiang Frank Wang1 Zhiding Yu1 Hengshuang Zhao2 Zhiqi Li1
1NVIDIA 2The University of Hong Kong 3MIT
Paper soon Code soon BibTeX Built on GR00T 2

Real robots

two Galaxea R1 Pro robots · private cameras only

Simulation · RoboFactory

67.8%
Real robots, three tasksmean success; 18.9 points above MA-VLA
90.3%
Simulation, eight tasksmean success with two to four arms
95.0% → 39.0%
Without cross-agent attentionfour-arm relay (Long Pipeline Delivery)
78.0% → 47.0%
Without team-stage future-video supervisionPlace Food

From a pair of robots in the lab to teams in factories and homes.

Abstract

Every move in a robot team changes the world in which its teammates must act. Coordinated manipulation therefore calls for predicting the team's actions together with the physical future they jointly create. We introduce MAWAM, a multi-agent world-action model that jointly generates each robot's future views and executable action chunks in a single sample. Built from a pretrained single-robot world-action backbone, MAWAM jointly denoises all robots' action and future-view tokens through cross-agent attention, while preserving a separate control interface for each robot. MAWAM achieves the highest success on all three real-robot tasks (laundry-bag collection, bath-towel folding, and umbrella handover), with 67.8% mean success, 18.9 percentage points above MA-VLA trained on the same demonstrations and cameras. In simulation, MAWAM reaches 90.3% mean success on eight tasks with teams of up to four arms. Ablations show that cross-agent attention and team-stage future-video supervision both improve coordinated control: without attention, a four-arm relay falls from 95.0% to 39.0%, and without this supervision, Place Food falls from 78.0% to 47.0%.

01 · Method

One model, the whole team's future.

Each robot keeps its own stream of private views, state and actions; an identity marks its views, and a role embedding marks all its tokens. One Transformer denoises every robot's future views and action chunk together, so each robot's actions are computed with its teammates' actions and the views they will produce. The robots execute the first 8 of 24 steps in lockstep, observe the new scene, and sample again.

TEAM SIZE

Readouts are the paper's simulation results for one task per team size (100 episodes per task). In simulation each arm has one private camera, and a read-only shared global camera adds context. Without attention, each agent keeps its own tokens and the shared global camera but cannot attend to its teammates' tokens; without team-stage future-video supervision, the video loss is removed in team training (Warm-up keeps it) but the future-view tokens stay.

PER-ROBOT STREAMS

Keep each output with its robot

A robot's private views, state and actions form one stream; a decoder returns each action chunk to that robot's controller.

IDENTITY + ROLE

Who acts, and what for

Simplex identities on a RoPE agent axis tie views to their robot, an action-to-video bias ties its actions to those views, and a learned role embedding marks its task function.

JOINT DENOISING

World and actions together

Flow matching on all private-view futures and action chunks, with dense attention across agents and prediction time.

TWO-STAGE TRAINING

Warm-up, then team training

Adapt the single-robot backbone one robot at a time, then learn from synchronized team demonstrations.

Method overview

1Warm-up

One robot at a time (same demonstrations)

From the pretrained backbone; roles, identity, cross-agent attention off

initialize

2Team training

Synchronized team demonstrations

TEAM TRAINING (a–c)

aPer-robot streams

  • Simulator view from Agent 1's private cameraIdentity

    Agent 1Camera holder 1

    • State
    • Role
    • Actions
    • Future views
  • Simulator view from Agent 2's private cameraIdentity

    Agent 2Camera holder 2

    • State
    • Role
    • Actions
    • Future views
  • Simulator view from Agent 3's private cameraIdentity

    Agent 3Meat presenter

    • State
    • Role
    • Actions
    • Future views

Identity: random simplex vertex on the RoPE agent axis

Simulator view from the shared global camera

Shared global camera (optional, read-only)

bCross-agent attention

Query A1 A2 A3 A1 A2 A3 Key / value

cJoint supervision

Video head
Action head
World-action objective (Eq. 2)

ℒWA = λvℒFM(O; Mv) + λuℒFM(U; Mu)

INFERENCE (SIMULATION)

dOne joint sample

  • Agent 1Camera holder 1

  • Agent 2Camera holder 2

  • Agent 3Meat presenter

Execute first 8 of 24 steps together → observe → sample again

Warm-up adapts the pretrained single-robot backbone one robot at a time; team training then learns from synchronized team demonstrations. In (a), a simplex identity on the RoPE agent axis marks each robot's views, and the arrow is the action-to-video bias, which adds the robot's action summary to its own future views. Shown: Camera Alignment in simulation, with one private view and a distinct role label per arm, plus a read-only shared global camera.
TWO-STAGE TRAINING

First one robot at a time, then the whole team.

The pretrained backbone is a single-robot world-action model. Warm-up adapts it to each robot's cameras and controller; team training then teaches it how the robots' actions affect one another. Both stages use the same world-action loss on future views and actions.

1Warm-up

Warm-up: One robot at a time

From
the pretrained single-robot world-action backbone
Sample
one robot's private views, state and actions (in simulation also the shared global camera, predicted like a private view); no cross-agent attention, identities or roles
Learns
each robot's cameras and controller
2Team training

Team training: The whole team together

From
the warm-up weights
Sample
a synchronized team demonstration, one stream per robot, with cross-agent attention, identities and roles; identities and robot order are randomized
Learns
how the robots' actions interact; every robot's future views and actions are supervised together

Why warm up first? Without warm-up, team training starts from the pretrained backbone, and success on the four-arm Long Pipeline Delivery falls from 95.0% to 34.0%. All tasks in the ablations

02 · Real robots

Two robots, six private cameras, no global view.

Two Galaxea R1 Pro robots, each with a head camera and two wrist cameras. Success rate over 30 trials per task; whiskers show the binomial standard error.

Real-robot success rate (%)

03 · Simulation

Teams of two to four arms.

Eight RoboFactory tasks with shared-object support, asymmetric transfers and ordered relays. MAWAM is evaluated on 100 episodes per task. Click a baseline in the legend to hide it.

Simulation success rate (%)

Hover or focus a row for all values.

04 · Ablations

What coordination needs.

Each ablation changes one factor of the full model; the Centralized team baseline instead treats the team as one robot. All are evaluated on the same 100 seeds per task. Pick a variant to compare it with full MAWAM on every task where it was run.

Full MAWAM vs. ablation (%)

MAWAM (full)ablation

05 · Predicted futures

What each robot expects to see next.

With every action chunk, MAWAM predicts each agent's private view up to 24 control steps ahead. Most of a view is static, so copying the last frame scores higher on full frames. On frames where at least 5% of the pixels change, the prediction is 4–7 dB closer to the executed frame at every horizon (Lift Barrier, Pass Shoe, Camera Alignment and Long Pipeline Delivery; 25 seeds each, with all 24 predicted steps executed before replanning).

Recorded and predicted views

Each replan shows 13 frames per view, at offsets of −12 to +24 control steps. The first five cover the observed context (in the bottom row, the model's reconstruction of it); the last eight are the future MAWAM predicted in its joint sample (bottom) next to what the camera recorded (top). In these clips the team replans every 8 steps, so recorded frames beyond +8 come after the next replan.

Single frames, six steps ahead

Camera Alignment

3 agents | shared global camera (read-only context)

Camera Alignment: shared global camera view at t (read-only context).

Shared global camera at t (read-only context, not predicted)

t | Input

t + 6 | Executed prefix

Observed

Model prediction

Executed rollout

Agent 1 Camera holder

Agent 1 (camera holder): observed private view at t.
Agent 1 (camera holder): MAWAM's predicted private view at t + 6.
Agent 1 (camera holder): executed private view at t + 6.

Agent 2 Camera holder

Agent 2 (camera holder): observed private view at t.
Agent 2 (camera holder): MAWAM's predicted private view at t + 6.
Agent 2 (camera holder): executed private view at t + 6.

Agent 3 Meat presenter

Agent 3 (meat presenter): observed private view at t.
Agent 3 (meat presenter): MAWAM's predicted private view at t + 6.
Agent 3 (meat presenter): executed private view at t + 6.

Additional coordination patterns

Matched private views at t + 6

Lift Barrier

2 agents | Agent 1 view Shared support

Model prediction

Lift Barrier, Agent 1 view: MAWAM's predicted private view at t + 6.

Executed rollout

Lift Barrier, Agent 1 view: executed private view at t + 6.

Pass Shoe

2 agents | Agent 2 view Object handover

Model prediction

Pass Shoe, Agent 2 view: MAWAM's predicted private view at t + 6.

Executed rollout

Pass Shoe, Agent 2 view: executed private view at t + 6.

Long Pipeline Delivery

4 agents | Agent 3 view Ordered relay

Model prediction

Long Pipeline Delivery, Agent 3 view: MAWAM's predicted private view at t + 6.

Executed rollout

Long Pipeline Delivery, Agent 3 view: executed private view at t + 6.
Predicted and executed observations. For each agent, the observed private view at t, MAWAM's prediction six control steps ahead from one joint sample, and the executed view at the same step.

When a teammate is held still

Seeds per receiver start delay (steps)
Delay (steps)MAWAMIndependent agents
< −602
−101
012
110
201
3014
4111
5417
6415
71115
81211
954
1023
1151
12140
13263
1470
1530
1610
1720
Response to a delayed teammate. In Long Pipeline Delivery, the giver of the second handover is held still for 16 control steps after it releases the shoe. MAWAM delays the receiver's start by a median of 12 steps; independent agents show a median delay of 6 steps. MAWAM's delay carries through to the grasp (median +9 vs. +1.5 steps) and is still 10 steps without the 18 episodes in which its giver re-grasps; under the hold, MAWAM succeeds in 99% of episodes vs. 44% for independent agents.
Citation

BibTeX

@misc{fan2026mawam,
  title   = {{MAWAM}: A Multi-Agent World-Action Model for Coordinated Manipulation},
  author  = {Fan, Xianzhe and Huang, Chi-Pin and Yang, Fu-En and Wang, Jing and Wu, Yecheng and
             Chou, Yu-Cheng and Bjorck, Johan and Fan, Linxi and Kautz, Jan and
             Wang, Yu-Chiang Frank and Yu, Zhiding and Zhao, Hengshuang and Li, Zhiqi},
  year    = {2026},
  url     = {https://research.nvidia.com/labs/lpr/mawam/}
}