Crucible

Two agents learn to talk, cooperate and use a tool, from scratch

The short version: neither agent is told to cooperate or talk. Both run one shared policy network and get a single team reward. Cooperation and communication show up only because the rooms make them necessary.

No pretrained models and no predefined symbol meanings. A procedural 3D world, built from composable rooms and simulated 8,192 times in parallel on one GPU, makes each skill necessary in turn. This report covers how it works and what the experiments show. The replay viewer shows the behavior.

Three headline claims

1. A communication protocol emerges and is causally necessary.

  • 99.9% success with the channel.
  • 24–26% when it's muted, shuffled or randomized, which is chance.
  • Independently trained pairs invent incompatible codes.
Watch with the channel muted

2. Cooperation without talking.

  • Hold needs both agents: 100% success, 0% if either one is idle.
  • Muting the channel changes nothing, so the agents coordinate by watching each other.
Watch with one agent idle

3. How objects are represented decides generalization.

  • A flat MLP binds behavior to an object's position in its input list and fails unseen combinations: 1.7% on hold+hold.
  • An entity-attention actor that sees objects as a set reaches 81–92%.
Compare on hold+hold

Plus a systems result: the simulator alone runs at 6.9M steps per second, but full training runs at 517k, because the learning step, not the simulator, is the bottleneck (4.5).

01The question

Don't reward the skill; require it. Build rooms that can't be passed without it, prove that with scripted agents, then test the trained agents by taking the skill away.

Can a pair of agents, trained only on a shared success signal, acquire qualitatively different skills (communicating, physically cooperating, using an object as a tool), and do those skills hold up in situations they never trained on?

Answering that cleanly took three design commitments:

  • Each skill must be necessary, not just helpful. If a shortcut exists, reinforcement learning will find it. Every room was checked with scripted agents before any training, to show that the intended skill is the only way through.
  • Evidence must be causal. Success rates alone don't show why agents succeed. So the trained agents are tested under interventions (mute the channel, idle a partner, move an object in the observation) and we measure what breaks.
  • Throughput decides how many questions you can ask. Thousands of worlds are simulated in parallel on the GPU, so a full run takes minutes to hours rather than days, and every claim can be re-run with more seeds.

02The world

Every room is checked for shortcuts before any agent trains in it. Five room types, composed into levels from Python, simulated 8,192 at a time on one GPU.

Madrona is a research game engine that runs thousands of independent worlds inside a single GPU program. World state lives in plain arrays (an entity-component system). Each simulation step is a fixed graph of small systems (movement, physics, buttons, doors, rewards, resets, observations). The arrays are shared directly with PyTorch, so observations never leave the GPU. Crucible forks Madrona's Escape Room example, which already provided rooms, doors, buttons, pushable cubes, a grab action and a renderer.

Python and PyTorch

  • Task sampler writes each world's room sequence
  • Actor: one shared recurrent policy
  • Critic: centralized, training only
  • Message channel and its interventions
  • Recurrent MAPPO update

Madrona simulator (C++/CUDA)

  • 8,192 worlds stepped together
  • Physics: 0.04 s ticks, 4 substeps
  • Doors, plates, buttons, rewards, auto-reset
  • State exported as GPU tensors, zero-copy
Figure 1. Simulation and learning share one GPU. The simulator's arrays are exposed directly as PyTorch tensors, so nothing is copied between the two halves.

What the fork changes

  • Composable levels. Each level is up to three rooms in a row, each room 13.3 × 20 units with its own door. Python writes each world's room sequence before it resets, so task mixtures, curricula and held-out splits change without recompiling the simulator.
  • A team reward, and no shortcuts. Both agents get +1 when the level is solved. The upstream reward for distance travelled and the "furthest progress" observation were removed, because either would let agents bypass the constraints being tested. The composition runs (4.4) add +0.2 per room cleared and +0.1 per room solved, reported as shaping.
  • Instrumentation. Per-episode task info and outcome are exported to Python. A separate export of exact positions feeds the viewer. A replay reset regenerates identical puzzles, so every checkpoint can be compared on the same levels.

Five rooms, each forcing one skill

RoomRuleWhat it forces
SignalFour pads in a jittered row, labelled in random order; one is the target. The seer is locked in a booth and is the only agent whose sensors flag the target. The doer must step on it. A wrong pad ends the episode.Communication (guessing scores 25%)Watch
HoldThe door is open only while a plate 7+ units away, or a return plate just past the door, is pressed. Doors close within about 4 steps, and agents move about 1 unit per step.Two agents working togetherWatch
CubeplateThe door is open only while its plate is pressed. There's no return plate, but there is a large pushable cube.Using the cube as a toolWatch
ButtonOne press opens the door for good.Warm-up
OpenThe door starts open.Warm-up and padding

Why the booth. Without it, the seer could walk over and stand on the target, or "point" with its body. Locked in a 2.6-unit booth with movement disabled (measured drift under random actions: 0.0005 units), the 8-symbol channel is the only route. The doer can't see which way the seer is facing either.

What the agents see and do

Observation: 145 numbers per agent, egocentric

Own state: position in room and level, heading, holding?, seer?, frozen?8
Partner: distance, angle, holding?3
Objects: 6 slots × (distance, angle, type, pressed, target flag)60
Door state3
Lidar: 30 rays60
Steps remaining1
Symbol heard last step (one-hot)8
Agent id2

The target flag is non-zero only in the seer's view. Agents act on these numbers, not pixels.

Action: five choices every step

Move distance4 options
Move direction8
Turn5
Grab / release2
Symbol to send (0 = silence)8

Episodes last up to 200 steps. The partner receives each symbol one step later, and every episode starts in silence.

Checked before any training

Scripted checkResultWhat it shows
Signal: doer that knows the target99.6%Solvable with perfect information
Signal: guessing doer25.0%Without information, chance is exactly 1 in 4
Signal: seer movement while locked in0.0005 unitsNo body-language side channel
Hold: scripted pair99.7%Solvable by two agents
Hold: best solo strategy0%Impossible alone
Cubeplate: scripted cube pusher23.6%Solvable with the cube, even by a crude script
Cubeplate: without the cube0.4%Essentially impossible without the tool

These checks caught a real bug: at first, a doer that knew the target scored only 68%, because walking straight at it brushed decoy pads. Pads now sit in a jittered row.

03How training works

Play, grade, learn, repeat. A critic that sees everything grades each decision during training, then leaves. At test time each agent acts only on what it can see.

The two agents play thousands of copies of the game at once. At first they act at random. Every few seconds training pauses to check which decisions turned out better than expected, and makes those decisions a little more likely. After a few hundred rounds, random flailing becomes a strategy, and in the Signal room that strategy includes a private language. MAPPO is the recipe for "make good decisions more likely" when two agents have to learn together.

Reinforcement learning in four ideas

Policy

The agent's instincts: given what it sees, a probability for every possible action. It's a neural network, and it outputs the five choices above every step.

Reward

The only feedback: +1 to both agents when the team solves the level. Nobody tells them how.

Trial and error

The policy is random to a degree, so agents naturally try different things. Learning means shifting probability toward what worked.

Credit assignment

The hard part. The reward arrives at the end, so which of the 20 to 200 earlier decisions earned it? Most of the machinery below exists to answer this well.

The loop: play, grade, learn

1Play

8,192 worlds × 64 steps with the current policy, frozen. Every decision is recorded: observation, action, its probability, the critic's prediction, reward, episode end.

≈ 1M agent decisions · 1–3 s

2Grade

For every decision: did things go better or worse than the critic expected? That number is the advantage.

milliseconds

3Learn

Make positive-advantage actions more likely and negative ones less, in small, clipped steps. Improve the critic. Then throw the data away.

2 passes × 4 minibatches · 1.5–2 s
Figure 2. One training iteration, repeated about 270 times for Signal (142M world steps). In the code this is the main loop of crucible/train.py: collect() plays and grades, update() learns. A world step is one tick of one world, which is two agent decisions.

Grading: the advantage

The critic is a second network whose only job is to predict future reward from the current situation. Think of a coach on the sideline saying "from here, I'd expect about a 25% chance of winning."

  • One-step surprise. Reward received + the coach's new prediction − the coach's old prediction. If the coach said 0.25 and one step later the doer is on the right pad (+1), the surprise is +0.75.
  • Advantage is the surprises from that point on, added up, with later ones counting less (this method is called GAE). Two dials control it:
    • γ = 0.998 sets how far ahead reward counts. At 0.99 (about a 100-step horizon) the doer learned to rush to a random pad: a quick 25% beat waiting. At 0.998 the horizon (about 500 steps) is longer than an episode, so waiting to hear the message isn't penalized.
    • λ = 0.95 trades real outcomes against the coach's guesses. λ = 1 is honest but noisy; λ = 0 is smooth but wrong whenever the coach is wrong. 0.95 is the standard compromise.
  • Why use a coach at all? The final score alone is noisy: you can play well and lose to bad luck. Subtracting a good prediction removes what the agent couldn't influence. A better coach gives a cleaner signal and faster learning. That is the whole reason for MAPPO's centralized critic, below.

Learning: the PPO update

  • Policy. For each recorded decision, raise the probability of the action taken if its advantage was positive and lower it if negative, in proportion to the size of the advantage.
  • Clipping, the "proximal" in PPO. The data was collected by the old policy. Change too much and the data no longer describes what the new policy would do. PPO tracks ratio = new probability ÷ old probability and stops pushing once it leaves 0.8–1.2. Small, safe steps.
  • Critic. It's trained to predict the returns that actually happened.
  • Entropy bonus. A small reward for keeping probabilities spread out, so agents keep exploring. It's smaller for the message head (0.002 vs 0.01) so symbols can become crisp.
loss = −mean( min(ratio·A, clip(ratio, 0.8, 1.2)·A) )  # policy
       + 0.5 · (V − observed return)²                  # critic
       − entropy bonus                                 # exploration

Plus the Adam optimizer at learning rate 3e-4, gradient clipping at 0.5, per-batch advantage normalization and running normalization of the critic's targets (ValueNorm), so scales stay stable as returns climb from about 0 to about 1.

What multi-agent adds

Plain PPO assumes one agent in a fixed world. With two learners, two things break. First, the target moves: to the doer, the seer is part of the environment, and it keeps changing as it learns. Second, credit is shared: both get the same +1, and the doer can't even see the target, so it can't tell a good move from a lucky one. MAPPO (Yu et al., 2022) is plain PPO plus three ingredients:

IngredientWhat it isWhy it helps
Shared policyOne actor network drives both agents. Each keeps its own memory and has an "am I the seer?" flag.Half the parameters and twice the data per parameter; basic skills like walking are shared.
Centralized criticThe critic sees both agents' observations plus hidden facts: room types, who is the seer, the target.Much sharper predictions, so cleaner advantages and faster, more stable learning.
Team rewardBoth agents get the same reward.No incentive to compete.

Centralized training, decentralized execution. The critic is used only to grade decisions during training. When the trained agents play, only the actor runs, on each agent's own observation. So letting the critic "cheat" is fine: it never acts. Example: the doer walks to pad 2. A critic without the target would predict "25%, as usual" and learn nothing. Ours knows the target is pad 3, so it predicts about 0, and that walk gets a strong negative advantage straight away.

The networks, and memory

Actor
own observation145→ LayerNorm + MLP2 × 256→ GRU memory256→ 5 output heads4 · 8 · 5 · 2 · 8
Critic
own + partner obs + task facts319→ LayerNorm + MLP2 × 256→ GRU memory256→ value1 training only
Variant
6 object tokens+ 1 context token→ transformer, no slot embeddingsd 128 · 4 heads · 2 layers→ average over objects→ replaces the actor's MLP
Figure 3. About 1.06M parameters in total (1.33M with the entity-attention actor). The actor's output heads read [current features, memory]. The entity-attention variant (4.4) sees objects as a set: with no slot embeddings it can't tell which list position an object came from.

Memory. The doer has to keep what it heard while it walks, and messages arrive a step late. During play each agent's GRU state carries forward and is wiped at every new episode. During learning, the stored steps are replayed in 32-step chunks, each starting from the memory saved during play (truncated backpropagation through time). Minibatches hold whole worlds, both agents' chunks together. That lets the critic's two-agent input be rebuilt on the fly instead of stored twice, and it makes the channel gradient below possible.

Where communication fits

To the learning algorithm, a message is just another action: the fifth output, eight options. The partner sees it next step as part of its observation. Nothing tells either agent what any symbol means. The channel lives in Python, so muting it, shuffling it across worlds, or randomizing it is a one-line intervention.

Why it's hard: chicken and egg. The doer's learning signal for "go to pad 3 when I hear symbol 6" averages to zero unless symbol 6 actually goes with pad 3, and at the start it doesn't. The seer's signal for "say 6 when the target is pad 3" averages to zero unless the doer responds to 6, and at the start it doesn't. Each side waits for the other until a small random correlation gets amplified. That is why runs sit at exactly 25% for tens of millions of steps and then jump (Figure 4).

DIAL: a gradient wire through the channel. During learning only, both agents of a world are replayed side by side, and the doer's learning signal flows backward through the message it received into the seer's choice of symbol. The received one-hot becomes stored + p − stopgrad(p), where p is the seer's softmax over symbols. Its forward value is exactly what was played, so PPO's ratios stay exact; only the gradient path is new. At play time, messages are still discrete symbols. Without this wire, the same setup is called RIAL, plain RL.

04Results

All evaluation numbers use world seeds never seen in training, with 4,000 to 20,000 test episodes per cell unless stated otherwise.

4.1Communication emerges, and the doer depends on it

Take the channel away and a 99.9% team drops to chance, while the seer keeps sending its code. Two independently trained pairs can't understand each other.

Channel gradient (DIAL), 3 seeds

Plain RL, 3 seeds

Figure 4. Plateau, then jump. Training success on Signal against world steps. Every run sits at chance for 50–90M steps, then climbs. Steps to reach 50% success: DIAL 58–68M, plain RL 58–94M, so there's no reliable difference at three seeds each. One plain-RL run held at 75% for about 50M steps (three pads named, one not) before resolving.
Table

Interventions on trained agents

SeedIntactMutedShuffledRandom
099.9%24.8%24.9%25.0%
199.8%25.0%24.1%24.2%
299.9%25.5%25.0%24.9%

10,000–20,000 episodes per cell. Shuffled delivers well-formed messages from a different world. The seer keeps sending about 2 bits of information about the target under every intervention; only the doer's access changes.

Cross-play: mixing partners

Seer fromDoer fromSuccess
same runsame run99.8–99.9%
seed 0seed 130.4%
seed 1seed 032.2%
seed 1seed 227.9%
seed 2seed 032.3%

Each pair invents its own code. A seer from one run and a doer from another barely beat chance.

One run, step by step

UpdateWorld stepsSuccessEpisode lengthBits in symbolsWhat's happening
10.5M24.8%400.00Random. Message entropy is 2.08 nats = ln 8: symbols are uniform noise.
2010M25.1%230.00The doer has learned to rush to some pad: fast, 1 in 4.
5026M0.4%1990.02A dip: the doer stops committing to any pad and episodes time out, the same dawdling the no-channel control shows.
10052M24.6%1040.13Back at chance. A faint correlation is forming.
15079M38.8%1410.75Breakout: the code and the listening grow together. Watch
16084M65.0%830.95
270142M99.8%15.51.45Solved: the seer names the target, the doer walks straight there. Watch

From the showcase run's training log (8,192 worlds, DIAL on), about 18 minutes of training. Each update is 524,288 world steps. Bits is the mutual information between target and symbol during training, out of a maximum of 2.

What the code means. At the final checkpoint in the viewer, pads 2 and 3 have their own symbols (2 and 6). Pads 0 and 1 are told apart by place rather than label: symbol 3 means "the left one of pads 0 and 1" 99–100% of the time, and anything else means the right one. Within that pair, symbols carry 0.55 bits about side and only 0.08 bits about label. So measuring against pad labels alone reads 1.49 of 2 bits even at 99.97% success. The seer also doesn't steer the doer: given the target, its symbol carries 0.006 bits about where the doer is. Codebook

Diagnosing the stall. When the first runs sat at 25%, feeding the doer the true target through the channel (an "oracle" seer) reached 99.9%. That separated the two possible failures: the listener was learnable, so the problem was getting both sides to agree on a code.

4.2Cooperation without talking

Both agents are necessary; words are not. Idle either agent and success goes from 100% to 0%. Mute the channel and nothing changes.

Hold was trained on a curriculum mix (20% Open, 30% Button, 50% Hold rooms) and learned abruptly: 1.6% Hold success at 31M steps, 35% at 37M, 97.6% at 42M, 100% from 47M. Watch at 37M

ConditionSuccessEpisodes
Both agents, channel on100%8,012
Channel muted100%8,024
Channel shuffled100%8,025
Agent A idle0%4,096
Agent B idle0%4,096

The idle-agent result shows both agents are needed: the scripted check showed the room is unsolvable alone, and the trained policy confirms it. The muted and shuffled results show the coordination isn't verbal: the agents time their swap by watching each other and the door.

Watch with one agent idle

4.3Tool use

Every success uses the tool. Without the cube the room is unsolvable (0.4%). With it, the trained pair succeeds 98.3% of the time.

In Cubeplate the agents learn to push the cube onto the plate and leave together. Training success rose from 1.3% at 10M steps to 41% at 73M, 71% at 84M and 97% at 126M. Evaluation at 147M: 98.3% (3,000 test episodes). The room has no return plate, so without the cube no strategy works. This is one seed, with no intervention beyond the scripted no-cube check. Watch at 84M

4.4Generalizing to unseen room combinations

Same skills, new arrangement: a network that reads objects by list position breaks (1.7%), and one that reads them as a set mostly doesn't (81–92%).

Setup. Training covered the five rooms alone plus eight two-room pairs (signal+button, signal+cubeplate, button+hold, hold+button, cubeplate+button, button+cubeplate, open+hold, hold+open). Testing used six combinations never seen in training. Both actors trained for 1B steps with the same recipe, shaping included.

CombinationFlat MLP, 1BAttention, 315MAttention, 1B
signaltrained73.5%99.3%99.95%
hold, cubeplatetrained100%100%100%
signal+holdheld out72.4%99.2%99.9%
cubeplate+holdheld out100%100%100%
hold+holdheld out1.7%92.1%81.4%
hold+cubeplateheld out30.0%29.2%57.5%
signal+hold+cubeplateheld out22.3%19.4%37.2%
button+hold+cubeplateheld out27.0%25.2%57.5%

About 4,000 test episodes per cell; one training seed per architecture. MLP on hold+hold   Attention on hold+hold

The mechanism. Hold's return plate sits just past its door, so it appears in the next room's object list. In training it always landed in slot 0 or 1. In unseen combinations it lands in slot 2, and the flat MLP, which learned "the plate is in slot 1", stops recognizing it.

  • Causal probe. On the MLP at 210M steps, moving the return plate back to its trained slot at test time raised rooms cleared on hold+cubeplate from 0.22 to 1.02. But that pushed the cube out of its trained slot, so the next room failed instead.
  • The fix. An entity-attention actor without slot embeddings treats objects as a set, so list position carries no meaning. It wins every held-out combination at the same budget.
  • Still open. hold+cubeplate remains the hardest case. Three explanations were ruled out: the cube pushed onto the wrong plate (1–2% of failures), the first agent through standing on the wrong plate (1–3% of its waiting time), and that agent chasing the cube (3–13%). More training helps the attention actor (29% → 58%) but not the MLP.
  • Not monotonic. The attention actor's hold+hold score fell from 92% at 315M steps to 81% at 1B: held-out performance can drift while training performance stays flat.

4.5Systems: where the time goes

The simulator isn't the bottleneck; learning is. Simulation takes 16–26% of training time, so the next speedups have to come from the learner.

Figure 5. Seconds of wall-clock per million world steps, full training loop, on Signal with 64-step rollouts and 2 epochs × 4 minibatches on an otherwise idle RTX PRO 6000. Simulation is 16–26% of the time with the flat MLP actor and 8% with entity attention.
Table
  • Simulator alone: up to 6.93M world steps/s (single Hold room, 32,768 worlds). Three-room levels cost about twice as much (3.28M/s for signal+hold+cubeplate).
  • Full training: 235k steps/s at 2,048 worlds, 517k at 8,192, 616k at 16,384. Throughput grows with batch size but sub-linearly, because the update grows with it.
  • Tradeoff: the entity-attention actor generalizes better (4.4) but trains 2.6× slower (198k steps/s).
  • Implication: more simulator speed would barely help. The next speedups are on the learner side: an asynchronous actor–learner split (IMPALA- or SEED-style with V-trace), CUDA graphs, and fewer epochs with larger batches.

05Assumptions that broke

What broke is part of the result. Each correction made a claim narrower and easier to defend.

  1. "A random pad layout is fine." A scripted doer that knew the target scored only 68%: walking straight at it brushed decoy pads. Pads now sit in a jittered row (99.6%). Caught by the scripted solvability check.
  2. "Resetting the reward on reset is harmless." The reset ran in the same step as the success and erased its reward: 25% success, zero return. Caught by logging success and return separately.
  3. "Default PPO will find the protocol quickly." At γ = 0.99 the doer learned to rush to a random pad in about 9 steps and the speaker stayed random. Raising γ and lowering the message entropy bonus removed the pull to rush, but runs still sat at chance for tens of millions of steps.
  4. "Plain RL never discovers the code." Wrong: I had stopped the runs too early. Every run sits at chance for 50–90M steps, then jumps. Caught by a matched long control run.
  5. "Gradients through the channel (DIAL) are needed." Not at three seeds: plain RL learns the code too, and DIAL isn't reliably faster. A positive-signalling bias (Eccles et al., 2019) made messages confident but didn't help the doer listen.
  6. "The code names pads." Partly: one code names two pads by label and splits the other two by position. Measuring information against labels alone under-reads it. Caught by conditioning on spatial layout.
  7. "Rooms that work alone will compose." The MLP tied behavior to list positions and failed unseen hold+hold (1.7%), while a set-based actor reached 81–92%. Per-room scripted checks don't cover compositions, and hold+cubeplate is still the hardest case.
  8. "More GPUs would help most." The account was capped at one GPU, and the learner, not the simulator, is the bottleneck, so a faster single GPU mattered more than more worlds.

06Limitations and next steps

The weakest link is seeds: generalization rests on one run per architecture. More seeds come before any new claim.

Statistical strength

The communication result has three seeds per condition. The generalization result has one seed per architecture, and that's the first thing to fix. Tool use is a single seed.

Composition coverage

Scripted checks verified single rooms, not combinations. A composition-level solvability check would show whether hold+cubeplate is hard for the agents or hard by construction.

Open-endedness

A fixed set of rooms eventually runs out of things to teach. The composable level builder is the step toward more: a curriculum that samples combinations by learning progress (for example, prioritized level replay), more room types and more agents.

Language

The learned code is a hybrid of labels and positions. Larger vocabularies, more targets and multi-step messages would test whether compositional structure appears.

Throughput

The learner is the bottleneck: decouple acting from learning, capture the update in CUDA graphs, and put compute into the representation where it pays off.

Observations

Structured observations made the slot-order failure possible to diagnose. Pixels from Madrona's batch renderer are the natural next step, and the renderer already works in this setup.

07Questions

How do you know it's really communication?

The seer is locked in a booth with no side channel (0.0005 units of drift, and the doer can't see its heading). The interventions are on the trained agents: muted, shuffled and random messages all drop the doer to chance while the seer keeps sending about 2 bits. And independently trained pairs can't understand each other.

Isn't DIAL cheating?

Training is centralized; execution is fully discrete and decentralized. And plain RL reaches the same protocol with more variance, so the channel gradient isn't required for the result.

Why one shared network for two different roles?

The role flag in the observation tells the network which job it has. Both roles reuse basic skills like moving and avoiding walls, and sharing doubles the data per parameter.

Why PPO, not Q-learning or value decomposition (QMIX)?

PPO handles a factorized five-part action space and recurrent memory naturally, and it's stable with very large parallel batches. MAPPO matches or beats value-decomposition methods on standard cooperative benchmarks. Replay-based off-policy learning also sits badly with a partner whose behavior keeps changing.

Isn't on-policy learning wasteful?

Each batch is used for eight gradient steps, then discarded. But data is cheap here: the simulator produces a million decisions in a second or two and is only 16–26% of training time. The cost is in learning, not in generating data.

Why a recurrent policy?

Messages arrive a step late and a learned code can be noisy from step to step, so the doer has to integrate over time and remember what it heard while it walks. A GRU is the cheapest way to do that.

Is the timeout handled correctly by the critic?

Episode ends are treated as terminal, with no bootstrapping past them. That's safe because "steps remaining" is part of the observation, so a timeout is predictable rather than a surprise cut-off.

What's the weakest part?

Generalization rests on one seed per architecture, hold+cubeplate is unexplained, and solvability was checked per room rather than per combination. Those are the first three things to address.

Appendix

Configuration

Worlds in parallel8,192, two agents each
Rollout64 steps → 524,288 world steps (≈ 1M agent decisions) per update
Backprop through time32-step chunks; memory reset at episode start
Passes per batch2 epochs × 4 minibatches, minibatches made of whole worlds
OptimizerAdam, learning rate 3e-4, eps 1e-5; gradient-norm clip 0.5; bf16 autocast
Discount / GAEγ = 0.998 (communication and composition runs; 0.99 default), λ = 0.95
PPOClip 0.2, value coefficient 0.5, entropy 0.01 (movement heads) and 0.002 (message head)
NormalizationPer-batch advantage normalization; debiased running normalization of value targets
ActorLayerNorm → MLP 2 × 256 → GRU 256 → [features, memory] → 5 heads (4, 8, 5, 2, 8)
Entity-attention actor6 object tokens + 1 context token, d = 128, 4 heads, 2 layers, no slot embeddings
CriticInput 319 (own 145 + partner 145 + 29 privileged) → MLP 2 × 256 → GRU 256 → value
Parameters1,058,748 (MLP actor) · 1,331,140 (entity attention)
Simulation0.04 s ticks, 4 physics substeps; episodes up to 200 steps; rooms 13.3 × 20 units
Channel8 symbols, 0 = silence, one-step delay, silent at episode start

Code map

crucible_env/src/Simulator fork: sim.cpp (systems and task graph), level_gen.cpp (rooms), types.hpp (components and exports)
crucible/envs.pyEnvironment adapter and the message channel with its interventions
crucible/model.pyActor, critic, entity attention, DIAL replay
crucible/ppo.pyRecurrent MAPPO: collection, GAE, update
crucible/train.pyTraining entry point
crucible/evaluate.pyInterventions, cross-play, idle agent, mutual information, slot probe
crucible/scripted.pySolvability and bypass checks
crucible/record.py, analyze_*.pyReplay recording and code analysis for the viewer
showcase/, report/The replay viewer and this report
# Signal with the channel gradient, as in Figure 4
python -m crucible.train --run-dir runs/signal_dial --task signal --num-worlds 8192 \
    --gamma 0.998 --msg-entropy-coef 0.002 --comm-grad

# Causal evaluation on held-out world seeds
python -m crucible.evaluate --run runs/signal_dial --modes on,off,shuffle,random

Glossary

AdvantageHow much better or worse things went after a decision than the critic expected.
Critic / valueA network that predicts future reward from the current situation. Used only in training.
CTDECentralized training, decentralized execution: the critic sees everything during training; at test time each agent acts on its own view.
GAEGeneralized advantage estimation: summing one-step surprises with decay λ.
BPTTBackpropagation through time: training the memory by replaying sequences.
RIAL / DIALLearning to communicate with plain RL, or with gradients routed through the channel during training (Foerster et al., 2016).
Entity attentionA transformer over object tokens. Without slot embeddings it's invariant to the order objects are listed in.
Mutual informationBits of information a symbol carries about the target. 2 bits fully identifies one of four pads.
Cross-playPairing agents from independently trained runs, to test whether their conventions are shared.

References

  1. Yu, Velu, Vinitsky, Gao, Wang, Bayen, Wu. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. NeurIPS Datasets and Benchmarks, 2022.
  2. Schulman, Wolski, Dhariwal, Radford, Klimov. Proximal Policy Optimization Algorithms. 2017.
  3. Schulman, Moritz, Levine, Jordan, Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR, 2016.
  4. Foerster, Assael, de Freitas, Whiteson. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. NeurIPS, 2016.
  5. Eccles, Bachrach, Lever, Lazaridou, Graepel. Biases for Emergent Communication in Multi-agent Reinforcement Learning. NeurIPS, 2019.
  6. Shacklett, Rosenzweig, Xie, Sarkar, Szot, Wijmans, Koltun, Batra, Fatahalian. An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation. SIGGRAPH, 2023.
  7. Jiang, Grefenstette, Rocktäschel. Prioritized Level Replay. ICML, 2021.
  8. Espeholt et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. ICML, 2018.