In 2023, Michael Ivanitskiy and his collaborators trained small transformers from scratch to solve mazes. Their paper, and a 2025 follow-on by Spies and his team, reported probe-readable representations of the maze topology inside the trained models, and causal interventions on those probes that changed the models' generated paths in the directions you would expect. Mazes have since become one of the cleaner test cases in the world-models literature: a small synthetic domain in which a transformer appears to develop an internal representation of the task and then use it.
Like Othello and chess, maze is an attractive domain for DST. The structure of the task is clean, the training data is synthetic, and the model can be probed directly. The DST-motivated question is narrower than 'is there a representation'. It is: when the trained model navigates a maze, is the behaviour best described as a topology-invariant graph computation, or as a token-mediated procedure that exploits the structured adjacency-list format the model has been trained on?
The structural reading: the model builds an internal graph of the maze and uses that graph to decide which moves to make. The probes recover the graph; interventions on the graph change the model's decisions; the model navigates because it knows the topology.
The DST reading: the model builds a token-mediated procedure that follows the listed edges of the adjacency list from the junction toward the target. The 'graph' the probes recover is the signature of training on synthetic mazes with structured edge descriptions. The model navigates because the tokens describe the navigation.
Both readings account for high solve rates and probe-readable maze content. The diagnostics below try to pull them apart.
For this work we trained six maze transformers from scratch, since the Spies models are not publicly available. Four are 5x5: Stan and Terry (same training data, different architectures - Stan with learned positional embeddings and 8 attention heads, Terry with RoPE and 4 heads), and a DFS model and a Prim model (same architecture, different training distributions - DFS mazes have long corridors and few turns, Prim mazes have short corridors and frequent turns). Two are 7x7: a 19M-parameter undertrained model (58% solve rate) and a 50M-parameter model trained on two million mazes (99%). All six use the same library, the same tokenisation, and the same training loop.
We start, as ever, with the probes. We trained linear probes on layer-zero activations to read the forty wall positions of the 5x5 maze. Self-probes recover the wall structure at around 71% accuracy. Above the majority-class baseline of 63.6%, but not by much.
Cross-probes drop further. A probe trained on Stan and applied to Terry reads at 51-52%; Terry to Stan, the same. Both below chance. The two models encode the same wall information in incompatible private formats.
An untrained network reads the walls at 65.5% against the trained 68.3%, over a 63.7% floor - the thin layer-0 wall signal is mostly carried by the adjacency list itself. See the random-net control.
There is a sharper way to read these numbers. The forty connections fall into two groups: 24 that are roughly 50/50 in the training data, and 16 that are around 80% present. A probe doing structural wall-reading should encode all forty connections at similar precision. Instead, the correlation between a connection's base rate and how much the self-probe improves over that base rate is -0.97. The probes encode the most information about the connections that are hardest to guess from base rate, and almost none about the easy ones. The model has allocated its representational capacity where the training loss pushed it, not where a uniform structural representation would have put it.
The layer-by-layer pattern is consistent with this. Self-probe accuracy improves with depth (Stan 70.8% at layer zero, 79.0% at layer three; Terry similar). Cross-probe accuracy peaks mid-network at around 55% and then falls back to 51%. The gap between them widens from 21pp at layer zero to 27pp at layer five. Deeper layers are more private, not more universal.
We can ask the question that mattered in Othello: is this privacy from training data, or from random initialisation? We extended the cross-probing to all four 5x5 models. The DFS model and Stan share their training distribution and differ only in initialisation. The DFS model and the Prim model share their initialisation pattern and differ in training distribution. Cross-probe accuracy: DFS to Stan, 53.8%; DFS to Prim, 54.1%. The same.
This is the maze version of the seed result from Othello. The privacy is not a property of the data the models saw. It is a property of the geometry each model happened to build during training.
The pattern from Othello and chess was that the model could not distinguish a structurally impossible move from a statistically unusual one. In maze we tested the same question. At five points along each maze's path we replaced the true next token with one of four options: a legal-common move, a legal-surprising move, a wall-violation, and a non-adjacent move. We measured next-token entropy.
The result is more complicated than in the other domains. Across the four 5x5 models, three (Stan, DFS, Prim) show lower entropy after a wall violation than after a legal-surprising move. Terry shows the opposite.
| Model | N mazes | Wall entropy | Surprising entropy | Delta |
|---|---|---|---|---|
| Stan | 177 | 0.109 | 0.272 | -0.163 |
| Terry | 177 | 0.174 | 0.125 | +0.049 |
| DFS | 782 | 0.270 | 0.557 | -0.287 |
| Prim | 691 | 0.196 | 0.363 | -0.167 |
A structural wall-violation detector would be expected to give a consistent signature across attention variants. Three of the four models do, but Terry doesn't. So the entropy on its own is hard to read.
What we ran next is more direct. Instead of asking what the entropy does after an illegal move, we asked what the model actually does next. For every wall-violation and every legal-surprising injection we looked at the argmax token and continued generating for five steps.
The result across all four 5x5 models is remarkably consistent. 98-99% of the time the model's chosen next token is legal from the cell the model was just forced through. 0% of the time it returns to where it should have been. The model accepts the injected position, updates its trajectory, and continues navigating from the wall it just walked through.
This is the diagnostic in plainer language. Whatever the entropy sign does, the behaviour is the same across all four models. None of them flags the violation; none of them tries to back out. The architecture changes how the model distributes probability after the injection, but the structural behaviour, what the model actually does next, is identical.
The mechanism behind the sign variation is the backtrack-availability asymmetry. After a legal-surprising injection the previous cell is still a connected neighbour, so backtracking is a high-probability option that competes with forward moves. After a wall violation the previous cell is wall-blocked, so backtracking would itself be illegal, and the distribution concentrates on forward moves only. Stan and the DFS/Prim family handle this concentration one way; Terry handles it another. Neither outcome is wall-detection.
Same test as Othello and chess: raise the temperature, count the illegal moves. Stan stays under 2% illegal at temperature one. Terry climbs gently to 3%. The DFS and Prim models behave similarly. No threshold, no discontinuity, no point at which the model shifts from legal to illegal.
At temperature zero the greedy path happens to be legal. As the distribution flattens, the tail surfaces. There is nothing upstream of sampling that enforces legality.
The piece of work the maze technical paper does most distinctively is at junctions: points where the maze branches into two paths, one leading to the target and one to a dead end. On its face this is the test that looks most like genuine graph reasoning.
We constructed 200 maze pairs where each maze contains a clean junction. The Prim model scores 88.2%; the DFS model scores 71.5%. Both well above chance. A first reading is that the model is doing multi-hop graph computation: looking down each branch, seeing where it leads, picking the one that reaches the target.
Both models score well above chance. The structural reading would attribute this to multi-hop graph computation. The information-horizon tests below pull the mechanism apart.
We worked through the obvious confounds one by one. Manhattan distance to the target: even on the subset where the correct branch is the further one, the Prim model still scores 86%. Node degree at the first cell of each branch: not predictive (AUC under 0.55). Adjacency-list proximity: not predictive. Subtree size, the size of each branch's tree: not predictive (AUC 0.49). On the junctions where the two subtree sizes are balanced, accuracy is 87%; the model's behaviour is unaffected by which side is larger.
So the question becomes: if the model is not using these features, what is it using?
The cleanest line of attack was not to ask what the model is using, but what it needs. We progressively truncated the adjacency list, keeping only the edges within k hops of the junction.
| k | Accuracy |
|---|---|
| 1 | 48.8% |
| 2 | 48.3% |
| 3 | 49.0% |
| 4 | 52.8% |
| full | 89.0% |
The model needs the complete adjacency list. Up to four hops out, it scores at chance. With the full list, it scores 89%. Padding the truncated list to full length with random edges from other mazes does nothing; the model is not confused by length, it needs the correct distant edges. Replacing the entire list with a different maze's list drops accuracy below chance (38.8%): the model follows whatever list it sees.
This narrows the question. The model uses the full list, but which part of it carries the signal? We isolated the chain of edges connecting the junction to the target through the correct branch. With just the chain plus padding, accuracy lands at 59%: real signal, but not sufficient.
The sharpest test came from scrambling. Starting from the full correct list we progressively replaced N edges with random ones, in three conditions: corrupt random edges, preserve the target-to-junction chain, or specifically disrupt the chain.
| N scrambled | Random | Preserve chain | Disrupt chain | Gap |
|---|---|---|---|---|
| 2 | 70.0% | 79.8% | 49.3% | +30.5pp |
| 4 | 62.5% | 72.5% | 46.3% | +26.2pp |
| 6 | 52.5% | 68.8% | 36.8% | +32.0pp |
| 8 | 51.2% | 65.8% | 35.0% | +30.8pp |
A 26-32pp gap between preserving and disrupting the chain, at every level of corruption. The chain is carrying the largest measured effect. Preserving it holds accuracy at 62-80% even with extensive non-chain corruption. Disrupting it drops accuracy below chance.
A simpler confound has to be ruled out before reading this. The chain edges contain the target cell name. Disrupting them removes the target from the adjacency list. Could the chain effect just be a target-cell lookup? We tested directly: removing only the target edge but keeping the rest of the chain gave 71.8%; removing the rest of the chain but keeping the target edge gave 35%. The target cell name on its own carries no signal. What the model needs is the connecting edges between junction and target.
The conclusive test was relabelling. Take the cells on the two branches and swap their labels in selected portions of the adjacency list. On a 5x5 grid you cannot swap labels arbitrarily without creating geometric impossibilities (cells that are not grid-neighbours appearing as connected), so we filtered to 81 mazes where every swap remained grid-adjacent.
| Condition | Accuracy (N=81) |
|---|---|
| Full correct | 96.3% |
| Swap labels in chain edges only | 0.0% |
| Swap labels in non-chain edges only | 87.7% |
| Swap labels everywhere | 0.0% |
Both diagnostic conditions sit at 0%. Whether the swap is in the chain alone or everywhere, the model follows the redirected chain to the wrong branch in every case. Relabelling cells off the chain has almost no effect.
Swapping cell labels in the chain edges drops accuracy from 96.3% to 0%. Every single time, the model follows the redirected chain to the wrong branch. Swapping labels in non-chain edges has almost no effect.
The diagnostic for topology-invariance follows. We constructed 200 maze pairs that share the same junction topology but use different cell labels. A graph-traversal algorithm should give the same answer on the same graph regardless of node labels. The model's success on two labellings of the same topology should be correlated.
It isn't. The within-pair correlation across the 200 pairs is -0.087, not significantly different from zero. The four-bin breakdown (both right, both incorrect, A only, B only) matches the distribution you would expect if the two outcomes were statistically independent. The significant predictor of success is chain length: shorter chains succeed more often than longer ones. Conditional on chain length, which specific filler edges sit where in the adjacency list seems to determine whether attention propagation along the chain succeeds on that particular instantiation.
A graph-traversal algorithm should give the same answer on identical topologies. The four-bin distribution here matches statistical independence: whether the model succeeds on maze A is uncorrelated with whether it succeeds on maze B.
The most natural way to read the chain corruption and the pair consistency together is as attention-mediated chain-following. Attention propagates along edges because the tokens describe edges: cell X appears in an entry with cell Y, so attending to X activates Y, and Y appears in an entry with Z. Across six layers, this propagation can follow the chain from target back to junction. The composition follows graph connectivity because the tokens describe graph connectivity.
At each layer, attention finds the edge token containing the current cell and reads the other cell from it. After three layers the activation has hopped from the junction along the chain to its end. Longer chains need more layers, which is the depth-limited gradient we see in the path-length data.
This explains both results at once. The chain matters because the propagation runs along it. Topology independence holds because the propagation depends on which specific tokens are present in which positions. Different cell-name tokens produce different attention patterns. Different attention patterns succeed or fail independently, even on the same underlying topology.
It is worth being precise about what this is and isn't. The model is doing real work: it is using multi-hop information from the adjacency list, and doing better than any one-hop heuristic could. But the work is not topology-invariant graph traversal. It is a token-mediated procedure that approximates traversal, with the approximation breaking down where the procedure runs out of token-level cues.
A simpler test confirms the mechanism from the other side. If the model follows the listed edges rather than computing topology, then replacing the adjacency list with a different maze's list should make the model follow the different list.
We did this directly. Take a test maze, keep the origin, target and path tokens, and swap in the adjacency list from a randomly selected DFS maze.
| Model | Follow rate | Acc when donor says through | Acc when donor says dead |
|---|---|---|---|
| DFS (Stan) | 94.0% | 98.5% | 11.8% |
| Prim (Terry) | 91.4% | 92.3% | 9.8% |
The model reads the adjacency list and follows it. It does not check the list against the path. It does not verify consistency. Removing the list entirely causes a large entropy spike (DFS +1.31 bits; Prim +0.72) and accuracy collapses to chance for Prim. The DFS model has a secondary heuristic on coordinate tokens that lets it score 71.5% without any list; Prim does not.
The obvious objection at 5x5 is that the mazes are small and the models small enough not to demand a topology-invariant procedure. We trained two 7x7 models to test it. The first (19M parameters, 200K training mazes) solves 58% of test mazes. The second (50M parameters, 2M training mazes) solves 99%.
Both 7x7 models reproduce the diagnostic pattern. The injection delta is negative; the correction analysis shows the same 92.5% legal-from-injected-cell and 0.7% backtrack. The temperature curve is smooth and the illegal rate stays near zero (0.04% to 0.16% across the temperature sweep). A 50M-parameter model that solves 99% of 7x7 mazes is showing similar distributional surface properties to the 19M-parameter 5x5 models. Better calibrated; same diagnostic shape.
One earlier piece of work in this battery did not pan out, and is worth flagging because it took a lot of time. We tried a signature-inversion construction analogous to the chess one: build mazes where the dead-end branch has the surface features of a through-route (high degree, many edges, branching structure) and the through-route has dead-end features (low degree, linear chain). The original results looked dramatic: the Prim model fell from 88.2% to 41.5%.
But the construction added cross-connections within the dead-end cluster to boost its degree, and those cross-connections created cycles. The training mazes are spanning trees, which means acyclic. The cyclic mazes were out-of-distribution. On the acyclic subset the effect disappears: the Prim model scores 92.5%, higher than baseline.
Several corrected acyclic constructions ran into a second confound: forcing the dead-end subtree to be larger compresses the through-route to a stub, and the through-route start becomes the target itself in over half the resulting mazes. This is a geometric constraint, not a construction bug. On spanning trees of the sizes tested (25 nodes at 5x5, 49 nodes at 7x7), the inversion cannot be cleanly executed.
The mechanism question was resolved by the chain corruption and pair consistency tests instead. The signature inversion attempt is in the technical paper for the record.
The maze experiments do not refute the Ivanitskiy or Spies findings. Probes do recover maze information in the models we trained, the recovery is causally related to the model's outputs, and the models do solve mazes. What the DST battery adds is calibration.
The probes are private. Each model encodes the maze in a geometry no other model can read, and the privacy comes from random initialisation, not from training distribution: this is the same pattern as the Othello seed result. Per-connection, the encoding tracks where the training loss gradient was steepest, not where a uniform structural representation would put it.
The injection entropy signature is architecture-dependent across four models that all behave the same way functionally. None of the four detects wall violations in the correction analysis. The entropy reflects how each model's attention distributes probability after the injection, not a violation detector.
The junction work is the most distinctive part of the maze battery. Forced-choice accuracy at junctions is high, but the high accuracy survives every one-hop and multi-hop confound we could construct, and is best explained by a mechanism that follows the chain of listed edges from junction to target through the correct branch. Chain corruption drives accuracy to zero. Pair consistency on same-topology different-label pairs is statistically independent. The mechanism that fits both results is attention-mediated chain-following: real multi-hop computation, but on tokens rather than on topology.
The 7x7 v2 model addresses the scale objection. A 50M-parameter model that solves 99% of 7x7 mazes shows the same diagnostic shape as the 19M-parameter 5x5 models.
Two readings remain available. A token-mediated chain-following reading is consistent with all of the evidence here. A weaker structural reading is also available: the models do use multi-hop information, the chain does matter, and a different architecture, or a curriculum that varies the labelling during training, or a much larger model, might surface a more topology-invariant procedure. None of these tests rules that out. What they do, taken together, is place the current evidence closer to a distributional map of synthetic-maze adjacency-list statistics than to a topology-invariant graph algorithm.