1. A communication protocol emerges and is causally necessary.
- 99.9% success with the channel.
- 24–26% when it's muted, shuffled or randomized, which is chance.
- Independently trained pairs invent incompatible codes.
The short version: neither agent is told to cooperate or talk. Both run one shared policy network and get a single team reward. Cooperation and communication show up only because the rooms make them necessary.
No pretrained models and no predefined symbol meanings. A procedural 3D world, built from composable rooms and simulated 8,192 times in parallel on one GPU, makes each skill necessary in turn. This report covers how it works and what the experiments show. The replay viewer shows the behavior.
Plus a systems result: the simulator alone runs at 6.9M steps per second, but full training runs at 517k, because the learning step, not the simulator, is the bottleneck (4.5).
Don't reward the skill; require it. Build rooms that can't be passed without it, prove that with scripted agents, then test the trained agents by taking the skill away.
Can a pair of agents, trained only on a shared success signal, acquire qualitatively different skills (communicating, physically cooperating, using an object as a tool), and do those skills hold up in situations they never trained on?
Answering that cleanly took three design commitments:
Every room is checked for shortcuts before any agent trains in it. Five room types, composed into levels from Python, simulated 8,192 at a time on one GPU.
Madrona is a research game engine that runs thousands of independent worlds inside a single GPU program. World state lives in plain arrays (an entity-component system). Each simulation step is a fixed graph of small systems (movement, physics, buttons, doors, rewards, resets, observations). The arrays are shared directly with PyTorch, so observations never leave the GPU. Crucible forks Madrona's Escape Room example, which already provided rooms, doors, buttons, pushable cubes, a grab action and a renderer.
| Room | Rule | What it forces | |
|---|---|---|---|
| Signal | Four pads in a jittered row, labelled in random order; one is the target. The seer is locked in a booth and is the only agent whose sensors flag the target. The doer must step on it. A wrong pad ends the episode. | Communication (guessing scores 25%) | Watch |
| Hold | The door is open only while a plate 7+ units away, or a return plate just past the door, is pressed. Doors close within about 4 steps, and agents move about 1 unit per step. | Two agents working together | Watch |
| Cubeplate | The door is open only while its plate is pressed. There's no return plate, but there is a large pushable cube. | Using the cube as a tool | Watch |
| Button | One press opens the door for good. | Warm-up | |
| Open | The door starts open. | Warm-up and padding |
Why the booth. Without it, the seer could walk over and stand on the target, or "point" with its body. Locked in a 2.6-unit booth with movement disabled (measured drift under random actions: 0.0005 units), the 8-symbol channel is the only route. The doer can't see which way the seer is facing either.
| Own state: position in room and level, heading, holding?, seer?, frozen? | 8 |
| Partner: distance, angle, holding? | 3 |
| Objects: 6 slots × (distance, angle, type, pressed, target flag) | 60 |
| Door state | 3 |
| Lidar: 30 rays | 60 |
| Steps remaining | 1 |
| Symbol heard last step (one-hot) | 8 |
| Agent id | 2 |
The target flag is non-zero only in the seer's view. Agents act on these numbers, not pixels.
| Move distance | 4 options |
| Move direction | 8 |
| Turn | 5 |
| Grab / release | 2 |
| Symbol to send (0 = silence) | 8 |
Episodes last up to 200 steps. The partner receives each symbol one step later, and every episode starts in silence.
| Scripted check | Result | What it shows |
|---|---|---|
| Signal: doer that knows the target | 99.6% | Solvable with perfect information |
| Signal: guessing doer | 25.0% | Without information, chance is exactly 1 in 4 |
| Signal: seer movement while locked in | 0.0005 units | No body-language side channel |
| Hold: scripted pair | 99.7% | Solvable by two agents |
| Hold: best solo strategy | 0% | Impossible alone |
| Cubeplate: scripted cube pusher | 23.6% | Solvable with the cube, even by a crude script |
| Cubeplate: without the cube | 0.4% | Essentially impossible without the tool |
These checks caught a real bug: at first, a doer that knew the target scored only 68%, because walking straight at it brushed decoy pads. Pads now sit in a jittered row.
Play, grade, learn, repeat. A critic that sees everything grades each decision during training, then leaves. At test time each agent acts only on what it can see.
The two agents play thousands of copies of the game at once. At first they act at random. Every few seconds training pauses to check which decisions turned out better than expected, and makes those decisions a little more likely. After a few hundred rounds, random flailing becomes a strategy, and in the Signal room that strategy includes a private language. MAPPO is the recipe for "make good decisions more likely" when two agents have to learn together.
The agent's instincts: given what it sees, a probability for every possible action. It's a neural network, and it outputs the five choices above every step.
The only feedback: +1 to both agents when the team solves the level. Nobody tells them how.
The policy is random to a degree, so agents naturally try different things. Learning means shifting probability toward what worked.
The hard part. The reward arrives at the end, so which of the 20 to 200 earlier decisions earned it? Most of the machinery below exists to answer this well.
8,192 worlds × 64 steps with the current policy, frozen. Every decision is recorded: observation, action, its probability, the critic's prediction, reward, episode end.
≈ 1M agent decisions · 1–3 sFor every decision: did things go better or worse than the critic expected? That number is the advantage.
millisecondsMake positive-advantage actions more likely and negative ones less, in small, clipped steps. Improve the critic. Then throw the data away.
2 passes × 4 minibatches · 1.5–2 scrucible/train.py: collect() plays and grades, update() learns. A world step is one tick of one world, which is two agent decisions.The critic is a second network whose only job is to predict future reward from the current situation. Think of a coach on the sideline saying "from here, I'd expect about a 25% chance of winning."
loss = −mean( min(ratio·A, clip(ratio, 0.8, 1.2)·A) ) # policy
+ 0.5 · (V − observed return)² # critic
− entropy bonus # exploration
Plus the Adam optimizer at learning rate 3e-4, gradient clipping at 0.5, per-batch advantage normalization and running normalization of the critic's targets (ValueNorm), so scales stay stable as returns climb from about 0 to about 1.
Plain PPO assumes one agent in a fixed world. With two learners, two things break. First, the target moves: to the doer, the seer is part of the environment, and it keeps changing as it learns. Second, credit is shared: both get the same +1, and the doer can't even see the target, so it can't tell a good move from a lucky one. MAPPO (Yu et al., 2022) is plain PPO plus three ingredients:
| Ingredient | What it is | Why it helps |
|---|---|---|
| Shared policy | One actor network drives both agents. Each keeps its own memory and has an "am I the seer?" flag. | Half the parameters and twice the data per parameter; basic skills like walking are shared. |
| Centralized critic | The critic sees both agents' observations plus hidden facts: room types, who is the seer, the target. | Much sharper predictions, so cleaner advantages and faster, more stable learning. |
| Team reward | Both agents get the same reward. | No incentive to compete. |
Centralized training, decentralized execution. The critic is used only to grade decisions during training. When the trained agents play, only the actor runs, on each agent's own observation. So letting the critic "cheat" is fine: it never acts. Example: the doer walks to pad 2. A critic without the target would predict "25%, as usual" and learn nothing. Ours knows the target is pad 3, so it predicts about 0, and that walk gets a strong negative advantage straight away.
Memory. The doer has to keep what it heard while it walks, and messages arrive a step late. During play each agent's GRU state carries forward and is wiped at every new episode. During learning, the stored steps are replayed in 32-step chunks, each starting from the memory saved during play (truncated backpropagation through time). Minibatches hold whole worlds, both agents' chunks together. That lets the critic's two-agent input be rebuilt on the fly instead of stored twice, and it makes the channel gradient below possible.
To the learning algorithm, a message is just another action: the fifth output, eight options. The partner sees it next step as part of its observation. Nothing tells either agent what any symbol means. The channel lives in Python, so muting it, shuffling it across worlds, or randomizing it is a one-line intervention.
Why it's hard: chicken and egg. The doer's learning signal for "go to pad 3 when I hear symbol 6" averages to zero unless symbol 6 actually goes with pad 3, and at the start it doesn't. The seer's signal for "say 6 when the target is pad 3" averages to zero unless the doer responds to 6, and at the start it doesn't. Each side waits for the other until a small random correlation gets amplified. That is why runs sit at exactly 25% for tens of millions of steps and then jump (Figure 4).
DIAL: a gradient wire through the channel. During learning only, both agents of a world are replayed side by side, and the doer's learning signal flows backward through the message it received into the seer's choice of symbol. The received one-hot becomes stored + p − stopgrad(p), where p is the seer's softmax over symbols. Its forward value is exactly what was played, so PPO's ratios stay exact; only the gradient path is new. At play time, messages are still discrete symbols. Without this wire, the same setup is called RIAL, plain RL.
All evaluation numbers use world seeds never seen in training, with 4,000 to 20,000 test episodes per cell unless stated otherwise.
Take the channel away and a 99.9% team drops to chance, while the seer keeps sending its code. Two independently trained pairs can't understand each other.
| Seed | Intact | Muted | Shuffled | Random |
|---|---|---|---|---|
| 0 | 99.9% | 24.8% | 24.9% | 25.0% |
| 1 | 99.8% | 25.0% | 24.1% | 24.2% |
| 2 | 99.9% | 25.5% | 25.0% | 24.9% |
10,000–20,000 episodes per cell. Shuffled delivers well-formed messages from a different world. The seer keeps sending about 2 bits of information about the target under every intervention; only the doer's access changes.
| Seer from | Doer from | Success |
|---|---|---|
| same run | same run | 99.8–99.9% |
| seed 0 | seed 1 | 30.4% |
| seed 1 | seed 0 | 32.2% |
| seed 1 | seed 2 | 27.9% |
| seed 2 | seed 0 | 32.3% |
Each pair invents its own code. A seer from one run and a doer from another barely beat chance.
| Update | World steps | Success | Episode length | Bits in symbols | What's happening |
|---|---|---|---|---|---|
| 1 | 0.5M | 24.8% | 40 | 0.00 | Random. Message entropy is 2.08 nats = ln 8: symbols are uniform noise. |
| 20 | 10M | 25.1% | 23 | 0.00 | The doer has learned to rush to some pad: fast, 1 in 4. |
| 50 | 26M | 0.4% | 199 | 0.02 | A dip: the doer stops committing to any pad and episodes time out, the same dawdling the no-channel control shows. |
| 100 | 52M | 24.6% | 104 | 0.13 | Back at chance. A faint correlation is forming. |
| 150 | 79M | 38.8% | 141 | 0.75 | Breakout: the code and the listening grow together. Watch |
| 160 | 84M | 65.0% | 83 | 0.95 | |
| 270 | 142M | 99.8% | 15.5 | 1.45 | Solved: the seer names the target, the doer walks straight there. Watch |
From the showcase run's training log (8,192 worlds, DIAL on), about 18 minutes of training. Each update is 524,288 world steps. Bits is the mutual information between target and symbol during training, out of a maximum of 2.
What the code means. At the final checkpoint in the viewer, pads 2 and 3 have their own symbols (2 and 6). Pads 0 and 1 are told apart by place rather than label: symbol 3 means "the left one of pads 0 and 1" 99–100% of the time, and anything else means the right one. Within that pair, symbols carry 0.55 bits about side and only 0.08 bits about label. So measuring against pad labels alone reads 1.49 of 2 bits even at 99.97% success. The seer also doesn't steer the doer: given the target, its symbol carries 0.006 bits about where the doer is. Codebook
Diagnosing the stall. When the first runs sat at 25%, feeding the doer the true target through the channel (an "oracle" seer) reached 99.9%. That separated the two possible failures: the listener was learnable, so the problem was getting both sides to agree on a code.
Both agents are necessary; words are not. Idle either agent and success goes from 100% to 0%. Mute the channel and nothing changes.
Hold was trained on a curriculum mix (20% Open, 30% Button, 50% Hold rooms) and learned abruptly: 1.6% Hold success at 31M steps, 35% at 37M, 97.6% at 42M, 100% from 47M. Watch at 37M
| Condition | Success | Episodes |
|---|---|---|
| Both agents, channel on | 100% | 8,012 |
| Channel muted | 100% | 8,024 |
| Channel shuffled | 100% | 8,025 |
| Agent A idle | 0% | 4,096 |
| Agent B idle | 0% | 4,096 |
The idle-agent result shows both agents are needed: the scripted check showed the room is unsolvable alone, and the trained policy confirms it. The muted and shuffled results show the coordination isn't verbal: the agents time their swap by watching each other and the door.
Every success uses the tool. Without the cube the room is unsolvable (0.4%). With it, the trained pair succeeds 98.3% of the time.
In Cubeplate the agents learn to push the cube onto the plate and leave together. Training success rose from 1.3% at 10M steps to 41% at 73M, 71% at 84M and 97% at 126M. Evaluation at 147M: 98.3% (3,000 test episodes). The room has no return plate, so without the cube no strategy works. This is one seed, with no intervention beyond the scripted no-cube check. Watch at 84M
Same skills, new arrangement: a network that reads objects by list position breaks (1.7%), and one that reads them as a set mostly doesn't (81–92%).
Setup. Training covered the five rooms alone plus eight two-room pairs (signal+button, signal+cubeplate, button+hold, hold+button, cubeplate+button, button+cubeplate, open+hold, hold+open). Testing used six combinations never seen in training. Both actors trained for 1B steps with the same recipe, shaping included.
| Combination | Flat MLP, 1B | Attention, 315M | Attention, 1B | |
|---|---|---|---|---|
| signal | trained | 73.5% | 99.3% | 99.95% |
| hold, cubeplate | trained | 100% | 100% | 100% |
| signal+hold | held out | 72.4% | 99.2% | 99.9% |
| cubeplate+hold | held out | 100% | 100% | 100% |
| hold+hold | held out | 1.7% | 92.1% | 81.4% |
| hold+cubeplate | held out | 30.0% | 29.2% | 57.5% |
| signal+hold+cubeplate | held out | 22.3% | 19.4% | 37.2% |
| button+hold+cubeplate | held out | 27.0% | 25.2% | 57.5% |
About 4,000 test episodes per cell; one training seed per architecture. MLP on hold+hold Attention on hold+hold
The mechanism. Hold's return plate sits just past its door, so it appears in the next room's object list. In training it always landed in slot 0 or 1. In unseen combinations it lands in slot 2, and the flat MLP, which learned "the plate is in slot 1", stops recognizing it.
The simulator isn't the bottleneck; learning is. Simulation takes 16–26% of training time, so the next speedups have to come from the learner.
What broke is part of the result. Each correction made a claim narrower and easier to defend.
The weakest link is seeds: generalization rests on one run per architecture. More seeds come before any new claim.
The communication result has three seeds per condition. The generalization result has one seed per architecture, and that's the first thing to fix. Tool use is a single seed.
Scripted checks verified single rooms, not combinations. A composition-level solvability check would show whether hold+cubeplate is hard for the agents or hard by construction.
A fixed set of rooms eventually runs out of things to teach. The composable level builder is the step toward more: a curriculum that samples combinations by learning progress (for example, prioritized level replay), more room types and more agents.
The learned code is a hybrid of labels and positions. Larger vocabularies, more targets and multi-step messages would test whether compositional structure appears.
The learner is the bottleneck: decouple acting from learning, capture the update in CUDA graphs, and put compute into the representation where it pays off.
Structured observations made the slot-order failure possible to diagnose. Pixels from Madrona's batch renderer are the natural next step, and the renderer already works in this setup.
The seer is locked in a booth with no side channel (0.0005 units of drift, and the doer can't see its heading). The interventions are on the trained agents: muted, shuffled and random messages all drop the doer to chance while the seer keeps sending about 2 bits. And independently trained pairs can't understand each other.
Training is centralized; execution is fully discrete and decentralized. And plain RL reaches the same protocol with more variance, so the channel gradient isn't required for the result.
The role flag in the observation tells the network which job it has. Both roles reuse basic skills like moving and avoiding walls, and sharing doubles the data per parameter.
PPO handles a factorized five-part action space and recurrent memory naturally, and it's stable with very large parallel batches. MAPPO matches or beats value-decomposition methods on standard cooperative benchmarks. Replay-based off-policy learning also sits badly with a partner whose behavior keeps changing.
Each batch is used for eight gradient steps, then discarded. But data is cheap here: the simulator produces a million decisions in a second or two and is only 16–26% of training time. The cost is in learning, not in generating data.
Messages arrive a step late and a learned code can be noisy from step to step, so the doer has to integrate over time and remember what it heard while it walks. A GRU is the cheapest way to do that.
Episode ends are treated as terminal, with no bootstrapping past them. That's safe because "steps remaining" is part of the observation, so a timeout is predictable rather than a surprise cut-off.
Generalization rests on one seed per architecture, hold+cubeplate is unexplained, and solvability was checked per room rather than per combination. Those are the first three things to address.
| Worlds in parallel | 8,192, two agents each |
| Rollout | 64 steps → 524,288 world steps (≈ 1M agent decisions) per update |
| Backprop through time | 32-step chunks; memory reset at episode start |
| Passes per batch | 2 epochs × 4 minibatches, minibatches made of whole worlds |
| Optimizer | Adam, learning rate 3e-4, eps 1e-5; gradient-norm clip 0.5; bf16 autocast |
| Discount / GAE | γ = 0.998 (communication and composition runs; 0.99 default), λ = 0.95 |
| PPO | Clip 0.2, value coefficient 0.5, entropy 0.01 (movement heads) and 0.002 (message head) |
| Normalization | Per-batch advantage normalization; debiased running normalization of value targets |
| Actor | LayerNorm → MLP 2 × 256 → GRU 256 → [features, memory] → 5 heads (4, 8, 5, 2, 8) |
| Entity-attention actor | 6 object tokens + 1 context token, d = 128, 4 heads, 2 layers, no slot embeddings |
| Critic | Input 319 (own 145 + partner 145 + 29 privileged) → MLP 2 × 256 → GRU 256 → value |
| Parameters | 1,058,748 (MLP actor) · 1,331,140 (entity attention) |
| Simulation | 0.04 s ticks, 4 physics substeps; episodes up to 200 steps; rooms 13.3 × 20 units |
| Channel | 8 symbols, 0 = silence, one-step delay, silent at episode start |
crucible_env/src/ | Simulator fork: sim.cpp (systems and task graph), level_gen.cpp (rooms), types.hpp (components and exports) |
crucible/envs.py | Environment adapter and the message channel with its interventions |
crucible/model.py | Actor, critic, entity attention, DIAL replay |
crucible/ppo.py | Recurrent MAPPO: collection, GAE, update |
crucible/train.py | Training entry point |
crucible/evaluate.py | Interventions, cross-play, idle agent, mutual information, slot probe |
crucible/scripted.py | Solvability and bypass checks |
crucible/record.py, analyze_*.py | Replay recording and code analysis for the viewer |
showcase/, report/ | The replay viewer and this report |
# Signal with the channel gradient, as in Figure 4
python -m crucible.train --run-dir runs/signal_dial --task signal --num-worlds 8192 \
--gamma 0.998 --msg-entropy-coef 0.002 --comm-grad
# Causal evaluation on held-out world seeds
python -m crucible.evaluate --run runs/signal_dial --modes on,off,shuffle,random
| Advantage | How much better or worse things went after a decision than the critic expected. |
| Critic / value | A network that predicts future reward from the current situation. Used only in training. |
| CTDE | Centralized training, decentralized execution: the critic sees everything during training; at test time each agent acts on its own view. |
| GAE | Generalized advantage estimation: summing one-step surprises with decay λ. |
| BPTT | Backpropagation through time: training the memory by replaying sequences. |
| RIAL / DIAL | Learning to communicate with plain RL, or with gradients routed through the channel during training (Foerster et al., 2016). |
| Entity attention | A transformer over object tokens. Without slot embeddings it's invariant to the order objects are listed in. |
| Mutual information | Bits of information a symbol carries about the target. 2 bits fully identifies one of four pads. |
| Cross-play | Pairing agents from independently trained runs, to test whether their conventions are shared. |