Crucible

Two agents learn to talk, cooperate and use a tool, from scratch

The short version: neither agent is told to cooperate or talk. Both run one shared policy network and get a single team reward. Cooperation and communication show up only because the rooms make them necessary.

No pretrained models and no predefined symbol meanings. A 3D world built from rooms, simulated 8,192 times in parallel on one GPU.

Each room makes one skill necessary

seer symbol doer

Signal communication

Only the seer knows which of four pads is the target, and it's locked in a booth. The only way to tell the doer is an 8-symbol channel.

Guessing scores 25%

plate return 7+ units

Hold cooperation

The door stays open only while someone stands on a plate 7+ units away. One agent holds it for the other, then they swap using a plate just past the door.

Best solo strategy: 0%

cube plate

Cubeplate tool use

The door stays open only while its plate is pressed, and there's no return plate. The way out is to push the cube onto it.

Without the cube: 0.4%

Every room is checked with scripted agents before any training, so the intended skill is the only way through. The numbers above are those scripted checks.

One shared network, one team reward

  • Each agent sees 145 numbers from its own point of view (partner, objects, doors, 30 lidar rays, the last symbol heard) and makes 5 choices every step: move distance, direction, turn, grab, and a symbol to send.
  • Both agents run the same network. A role flag tells it whether it's the seer, and a GRU gives it memory.
  • A symbol is just another action. The partner hears it one step later, and nothing defines what it means.
  • The only reward is +1 to both agents when the level is solved.
  • During training, a critic that sees everything grades each decision. At test time it's gone, and each agent acts on its own view.

Training: play, grade, learn, repeat

  • The current policy plays the game in the simulator for rollout_len = 64 steps. It does this in every one of the parallel worlds at once (--num-worlds 8192 on one GPU, simulated by Madrona on the GPU).
  • Each step is recorded: observations, actions, rewards and value estimates. One rollout is 8192 × 64 ≈ 524k world steps.
  • GAE then works out how much better each action was than expected (the "advantages").
  • The update follows: the model trains on that batch for 2 epochs × 4 minibatches, which is 8 gradient steps.
  • PPO is on-policy, so the batch is thrown away and the next rollout starts with the improved policy. The N in update=N in the log counts these rollout→update cycles.

Signal took about 270 cycles: 142M world steps, about 18 minutes on one GPU.

1.A communication protocol emerges and is causally necessary.

  • 99.9% success with the channel.
  • 24–26% when it's muted, shuffled or randomized, which is chance.
  • Independently trained pairs invent incompatible codes.
Watch with the channel muted
chance (25%)

Signal success on levels never seen in training, range across 3 training seeds. The last bar pairs a seer and a doer from different runs.

The code appears suddenly, not gradually

  • Every run sits at chance for 50–90M world steps, then climbs.
  • First the doer learns to rush to any pad: quick, but right 1 time in 4.
  • Speaking and listening only pay off together. Each side waits for the other until a small random correlation gets amplified.
  • Plain RL finds the code too. Sending gradients through the channel (DIAL) isn't reliably faster at three seeds.
channel gradient (DIAL)plain RLchance (25%)

Training success on Signal against world steps, three seeds each.

2.Cooperation without talking.

  • Hold needs both agents: 100% success, 0% if either one is idle.
  • Muting the channel changes nothing, so the agents coordinate by watching each other.

Tool use is learned the same way: in Cubeplate the trained pair succeeds 98.3% of the time, and every success pushes the cube onto the plate.

Watch with one agent idle

Hold success on levels never seen in training, 4,096–8,025 episodes per bar.

3.How objects are represented decides generalization.

  • A flat MLP binds behavior to an object's position in its input list and fails unseen combinations: 1.7% on hold+hold.
  • An entity-attention actor that sees objects as a set reaches 81–92%.

Why: in every training level, Hold's return plate landed in list slot 0 or 1. In hold+hold it lands in slot 2, and the MLP stops recognizing it.

Compare on hold+hold

Success on hold+hold, a combination never seen in training. One seed per architecture, about 4,000 episodes per bar.

The simulator isn't the bottleneck; learning is.

  • The simulator alone runs at up to 6.9M world steps per second.
  • Full training runs at 517k per second with 8,192 worlds. Simulation is only 16–26% of the time.
  • So a faster simulator would barely help. The next speedups are on the learner side: an asynchronous actor–learner split, CUDA graphs, fewer epochs with larger batches.
6.9M

world steps per second, simulator alone

517k

world steps per second, full training loop

What's still weak

  • Seeds. Generalization rests on one run per architecture, and tool use on one run. More seeds come before any new claim.
  • hold+cubeplate is the hardest unseen combination (30% for the MLP, 58% for attention), and three explanations have already been ruled out.
  • Solvability was checked per room, not per combination, so hold+cubeplate may be hard by construction rather than hard for the agents.
  • Next: a curriculum that samples combinations by learning progress, more room types, and pixels instead of structured observations.

Summary

99.9% → 25%

Mute, shuffle or randomize the channel and the team drops to chance.

100% → 0%

Idle either agent and Hold becomes impossible. Muting changes nothing.

1.7% vs 81–92%

Unseen hold+hold: a flat MLP against an actor that sees objects as a set.