Othello

When the field investigated the internals of Othello-GPT, they did so with what I've called the Helmholtz expectation: that a sufficiently trained system builds an internal model of reality, extrapolating structural understanding from the data. This expectation is built on an important elision: that data, the residue produced by experience, contains the information required to rebuild experience itself.

In the first essay I laid out a guiding metaphor, which suggests the opposite: the difference between the landscape and the map. It says: the residue of experience, the data generated by it, is never enough to rebuild experience itself. The map may carry traces of the landscape, or even in places feel indistinguishable from it, but they are never the same thing.

So, how do we take this guiding metaphor and use it to try to investigate Othello-GPT? Can the metaphor account for what the researchers working with the model discovered?

Li et al. said that they had found internal board states in Othello-GPT, something analogous to an internal, spatial, representation of the game while the model was playing. This is the equivalent of a structural understanding.

The map/landscape distinction suggests something else: instead of building an internal model of reality, what if the model had learned a complete representation of its training distribution, a perfect map, and never built a landscape at all? This would be sophisticated navigation of a statistical surface.

Both hypotheses account for the existing evidence. The tools used to investigate Othello-GPT find the board state either way. The question is what that board state is.

If training on Othello leads to the emergence of a world model (WM), then different training distributions shouldn't matter; all runs should lead to a consistent internal representation of the rules that causally impact output.

If instead training is not the construction of a WM, but the process of creating and refining a map of the training data, then training multiple models should help us to see the unique signatures of their individual maps: the signature of their training data. With this, we can stress test whether what's underneath is a genuine structural understanding.

So, the fundamental question is: are Othello-GPT models building a structural representation of the game with behaviour preferences on top, or, perfect representations of the training data that includes the signatures of the game's structure in them?

To begin untangling these two, let's train three different types of models, at two different scales.

We now have 6 models we've trained, plus the pre-trained Othello-GPT from Li et al's original experiment - 7 Othello-GPTs.

Our 6 are trained on data we described as aggressive, uniform and passive. 3 have been trained on 500,000 synthetic games, and 3 have been trained on 20,000,000 synthetic games. These will show us how differences in scale of training affect behaviour.

Let's start by looking at how they behave in different game states, what moves they 'prefer', comparing the dense models (20m games) for aggressive and passive play with Li et al's pretrained model.

What characterises the 'behaviour' of each of the models is principally their training distribution - they 'play' the moves they've seen in their training. The aggressive model puts all its playing preference on high capture squares, the passive model on low capture moves. The uniform model, Li et al's original, spreads probability evenly across the 7 potential moves in the example above.

Whether you believe in the WM or map hypothesis, it's not a surprise that behaviour will be shaped by the training distribution. These were not models trained to 'win', just models trained to play.

But the extent is interesting: there is no convergence toward shared preferences, no sign that these dense models with 20,000,000 training games have produced anything beyond a more precise version of what the training data already contained. If at some point in training or behaviour a WM emerges, where would we see it?

A useful starting point is understanding what's happening to models as they train on more games. What are the differences between those trained on 500,000 games and 20,000,000?

As the model's capacity to accurately play Othello increases with the scale of training, do we see any emergent behaviours?

Increasing the scale of training data by 40x does only one thing: it solidifies the behavioural preferences already captured in the data.

More training means a more faithful map of the training data.

But again, reasonably, a person arguing for the WM hypothesis could say: the increased training builds out structural understanding. This is what enables the model to express the strategic biases inside its training data, of course behaviour gets sharper.

The world model in this formulation becomes two things: first, the board itself, the constraint on accuracy; and second, the structural understanding of the 'game', the projection of the rules onto that board. A WM can accommodate this result.

When we look more closely at the data, filtering for the places where board 'knowledge' converges, what do we see?

The same pattern. The data does show that board knowledge and distributional preference do not behave independently. Knowledge and preference move together and strengthen together. At every level of measurement the training distribution and the model's behaviour become more closely linked. The map becomes sharper, the behavioural preference becomes more entrenched, any structural features (accuracy of play, for example) just come along for the ride.

Though again the different models' deep commitment to their stylistic preferences doesn't negate a structural backdrop.

But right now we are still looking at what the models do, not what's inside them. If there's a world model buried under the behaviour, then it's the model's internal representations that should show it. If this WM is what is described above, an internal representation of the board and the rules, then those belong to the game of Othello, not each model's particular training experience. Shouldn't it, to some degree, be shared?

The probes that Li and Nanda used to 'find' the board state inside Othello-GPT are small neural nets trained to read a model's activations. When the probe succeeds, it tells us the information we trained the probe on is encoded in the geometry of the model's internal representations.

A probe establishes that a property is decodable from the activations. Decodable is not the same as used: the model may never consult what the probe can read. And decodable is not the same as learned either: the probe may be reading something the input carried in with it. The three collapse into each other very easily, and finding a way to untangle them is important to understand what the instruments used in this kind of mechanistic interpretability work are really doing.

So, we train probes on each of our seven models. Using nonlinear probes (the same type Li et al. used), board state recovery ranges from 95% to 99% accuracy depending on the model. The pretrained model reaches 99.5%. The information is there, encoded with remarkable fidelity, across every model. This replicates and confirms the original and follow up study's core finding.

But that recoverability obfuscated an important question: are the probes recovering a shared structural representation of Othello (the board, the rules and constraints) or a private encoding of what looks like the rules specific to each model? If the board state belongs to the game, if it is structural, then a probe trained on one model should be able to read another model's internals. The board is the board and it shouldn't matter which model encoded it if the models are converging towards something like structural understanding.

So, to test this, we took each model's probes and applied them to every other model's activations, taking 5,000 test positions of Othello.

Every probe fails on every other model. Even the two models that share the most similar training regime, the pretrained model and the dense uniform, cannot read each other. Cross-probe accuracy lands at chance: a mean of 33.3%, sitting right on the 33.3% floor of a three-way guess (the probe sorts each square into black, white or empty), and every one of the 90 cross-pairs sits between 31.7% and 34.6%.

Kindly, a researcher who audited this work pointed out to me that a probe is not a neutral instrument; it is a coordinate system, fitted to one model's internal geometry. Two networks could hold exactly the same information arranged at different angles, and every raw cross-probe would still fail. Failure to transfer shows that the coordinates differ, but it doesn't necessarily show that the content does. So if we rotate the coordinates, perhaps we find much more shared structure.

To run that test on the cleanest possible pair, we trained three more models: Seed A, Seed B and Seed C. They share the same architecture, the same training distribution, identical hyperparameters and the same training time. The only thing that differs is the random seed: the starting point of the learning process. If any two models should converge on a shared representation, it is these.

For each pair of models we learned a single rigid map between their activation spaces (a rotation plus one overall scaling, fitted on training positions and applied to held-out ones) and then asked the probes again.

One rotation recovers almost everything. Between two models trained on identical data from different seeds, the aligned probes read the board at 87.8% against a self-probe ceiling of 88.6% across the sixty playable squares: ninety-eight percent of the gap, closed by a change of coordinates. The dark matrix above is not a picture of private worlds. It is the same map, turned to different angles.

The privacy reading seems, then, simply incorrect. There was a shared board all along. But across different work in neuroscience, SAEs and other mechanistic interpretability literature, there have consistently been discoveries that probes may not be picking up what we think they are. So, to disambiguate, we can take a model with the same architecture that has learned nothing at all (randomly initialised, never trained on a single game) and run the identical pipeline on it.

The untrained network yields the board at 84.5% accuracy. The trained models reach 88.6%, over a majority-class baseline of 68.8% - the score you get by guessing the most common state of every square, which sits high here precisely because most of the board is trivially predictable. A network that has never seen a game of Othello already carries four fifths of the above-baseline board signal that full training produces, and the rotation works on it too: the same alignment procedure recovers ninety-three percent of the gap between two untrained networks. Probe success, cross-probe failure, alignment success, every one of these measurements is dominated by something that was present before any learning happened.

The board is a deterministic function of the move sequence, and an untrained transformer is a fixed random scrambling of its input. A linear probe is exactly the instrument that can read through the scrambling: what it found was never the model's possession, it came in with the data, because the data was generated by the game.

Training contributes something, since the trained models did read the board a few points better than the untrained ones. Everything else the probe reports is the game's residue in the data, recovered through whichever coordinates a particular run happened to settle into. So it's decodable, but most of it isn't 'learned'. Because of this problem with the methodology itself, the probe certainly cannot tell us if it's 'used'. Instead, we should look at the model's behaviour.

So we stop reading activations, and start watching what the models do, and those same seed models are the right place to begin.

Three models, identical in everything except the starting point of the learning process; the same map held at different angles. What do they do?

Across 5,000 Othello test positions they are almost indistinguishable. The KL divergence between their move preferences is 0.0007 - effectively zero. For practical purposes they are the same player.

The comparison becomes interesting at the next two tiers. Models trained on the same distribution but different amounts of data behave 190 times more differently, but models trained on different distributions behave 4,800 times more differently.

The random seed sets the angle at which a model stores what the data carries, and contributes almost nothing to what it does. Data volume sharpens behaviour and the training distribution determines it. Back in the metaphor of the map, this is what you would expect: you can draw a map many different ways - there is no obligation for the drawing to be universal - but what the map records, and where it takes you, is fixed by the landscape's residue in the data.

The original finding inferred outward from the inside of the model: a board in the activations, therefore a world model, therefore the behaviour follows. Every measurement in this essay has pushed the other way. The inside turned out to be the wrong place to ask the question, because the probes read the data through the model, and the coordinates are incidental. What we observe in Othello-GPT is the behaviour, on the surface the training data built. So, to really understand what the model has learned, and if they have a 'board' in the sense that it would matter for a WM, it should show up when they play.

The strongest formulation of the WM argument doesn't necessitate that an internal representation of Othello be shared across different models. As above:

The 'world model' is two things: first, the board itself, the constraint on accuracy; and second, the structural understanding of the 'game', the projection of the rules onto that board.

If we can't agree on whether the WM is intelligible in the models, or by inference between models, then we should be able to see it in the 'output'. How do these models play?

Let's take two of the seed models and the original pre-trained model. These models hold their boards in different coordinates but are almost identical in their behaviour. Let's have them play Othello. Let's bring them the board.

First let's let them play themselves: Seed A vs Seed A; Seed B vs Seed B etc. Two instances of the same model exchanging move tokens. If the model has an internal board that constrains its output, self-play should produce legal games. The board is the board; even at higher temperatures, which changes the sampling distribution, the constraint of a world model should ensure basic legality.

At temperature 0, where the model always picks its most confident prediction, all three models produce perfect legal play. At any temperature above zero, illegality appears. At temperature 1.0, about 8% of games contain at least one illegal move. The three independently trained models fail at indistinguishable rates. The curve is smooth. There is no threshold, no temperature at which the model shifts from structured play to breakdown.

There is a reasonable objection to this finding. Temperature is applied after the model's computation, it flattens the probability distribution, pushing mass from likely tokens onto unlikely ones. The WM, if it exists, operates upstream, it shapes the distribution before temperature corrupts it. So of course illegal moves surface at higher temperatures. The understanding is intact and the sampling is noisy.

But if the model has a WM that computes legality from the board state, why does it assign any probability to illegal moves at all?

Nothing in the architecture prevents the model from driving the logits of illegal tokens arbitrarily low. Yes, the probability mass on illegal squares is small, but it is present at every position, in every model. And it's structured: it correlates with how statistically common those squares are in the training distribution, not with how structurally illegal they are.

Temperature doesn't create this probability. It reveals probability that was already there; a map of training data. The map is a statistical surface where, over the training process, legality emerges, because every game the model trained on was legal.

But let's let the models play with each other. If what the models are really doing is coordinating shared surface statistics, then cross-play should match self-play exactly, because the statistical surface is the same regardless of which model is generating the tokens.

Cross-play is indistinguishable from self-play. The per-move illegal rate at every temperature is effectively identical. Models whose internal representations are mutually opaque coordinate just as well with strangers as with themselves. The delta across every temperature is noise.

The coordination doesn't come from a shared board. It comes from the fact that both models learned the same distribution. The distribution determines which tokens are probable. The probable tokens are overwhelmingly legal. The models agree on what to do because they agree on the surface, not because they share an internal model of the rules.

In AI when a model takes a 'greedy path' it means it picks its most confident next token (i.e. the token to which it has applied the greatest probability).

We've seen how at T=0 Othello-GPT models play a single deterministic game, but that game is always legal. This looks like evidence for the WM: left to its own devices, not thrown off by temperature, Othello-GPT plays legally.

To stress test, we decided to see how solid T=0 really was.

The setup is simple: an external Othello engine plays K number of random legal moves. So at K=0 Othello-GPT is just producing a token for a standard starting board, and K=1 Othello-GPT takes over after a first legal move has been made. We can increase K to put Othello-GPT in the early, mid or end part of a game.

The model's first move from an unfamiliar position is always legal. Across all prefix lengths and all three models, 0 out of 44,985 first moves were illegal. But the continuation wasn't.

The model's greedy play produces illegality in about 3% of games. The per-move illegal rate settles at roughly 0.17%, and it's flat from K=3 all the way through K=30. It doesn't matter how unfamiliar the starting position is, what's important is how many greedy steps the model takes. Each step has a small chance of being an illegal move and across a 40-move continuation, that chance accumulates.

T=0 legality from the model's own greedy path is not evidence of structural understanding. It is evidence that the single most practiced route through sequence space happens to be legal. It looks like a very well-trodden map, not a complex emergent world model.

But perhaps the best evidence for the World Model is not what's inside the model, or how the model behaves during gameplay, but how it responds to being perturbed.

The original finding in Li et al.'s paper was that the world model was not just observable but causal. When the researchers modified Othello-GPT's internal representation of a square, flipping it from say black to white, the model's predictions changed consistently with the new board state. It seemed to be using its internal representation of the board to course correct.

But we can go further: the signature of a WM, of structural understanding, would not be just 'whether' the model can respond to perturbation, but what it thinks about the perturbation itself. For example, an unusual move and an illegal move, for a model with a representation of the board in its head, should be 'experienced' differently.

To test this we took 100 legal Othello games and played them through to various points, from the early game through to late midgame. At each of the points we injected a move; these moves fell into four categories: a legal move that is statistically common at that stage, a legal move that is statistically rare, an illegal move to an empty square that captures nothing, and an illegal move to an already occupied square.

Then we measured the model's immediate response: the entropy (roughly the 'level of confusion') of its output distribution for the very next token.

The question was: with a coherent internal WM structure of the board, would Othello-GPT react differently to these perturbations?

No. The mean entropy after an illegal injection was 2.262. The mean entropy after a legal but surprising injection was 2.335. The model is slightly more uncertain after the legal surprise move than after the structurally impossible move; but the difference is tiny: 3.1%.

Across 1,800 measurements, three independently trained models, six injection points, the pattern is almost always the same. The model cannot really tell the difference.

If these models really had an internally coherent representation of the board state, one that causally impacted their output choices, then a truly illegal move should have a significantly more entropic response. A WM should have something like surprise baked in.

But on a distributional surface, there is no category error. There are only familiar contexts and unfamiliar contexts. A move to an occupied square is unfamiliar because it never appeared in training. A legal but rare move is unfamiliar because it rarely appeared in training. Both are tail events on the statistical surface.

The distinction between impossible and unusual is a property of the rules, not the statistics. The rules know that an occupied square cannot be played. The statistics only know that it wasn't played. These are different things, and the model has the second, not the first.

Data carries the signature of the world that generated it. And the map built from that data can carry it too, but they are not the same thing. The map is not the landscape.

There was one aspect of doing the Othello-GPT work that haunted me: what was 'the board'?

The board state, the causal impact of the board, board accuracy, this was the language of the papers and the language used to describe the model. But what did 'the board' mean to the Othello-GPT?

The best way to understand this might be to break it. We can inject illegal moves into an otherwise legal game, and track the internal representation that Othello-GPT has of the board using probes. For example: if someone places a piece on a square that already has a piece on it, that's not a legal move. We know already that the model doesn't treat this as any more or less likely than a surprising legal move, but does its representation change?

The true board state hasn't changed in the way a normal move would change it. If the model is maintaining a real representation of the board, the probes should still read the true board: the one where that square is still occupied by whoever was there before.

If the model doesn't have a board, if it's just predicting the most likely next state given what it's seen, then it will do what it always does: pattern-complete. It will generate the board state that would exist if the illegal move had been legal, as if the piece had been legitimately placed and the normal flipping rules applied. A board that makes internal sense but never actually existed.

Consistently the probes track the predicted board, not the 'true' one. Across all seven models, the board state the probes recovered after an illegal injection is the board that would exist if the illegal move were legal.

What does a board 'mean', anyway, if we know all this?

In the first essay I described the act of 'decompression': the cognitive process by which we restore the three-dimensional landscape of experience from the flat surface of words. The act of decompression is so fundamental to human communication that we cannot receive language without doing it. When we encounter the outputs of LLMs we perform this restoration, giving those outputs dimensions they never had to begin with.

In Othello-GPT, the decompression was not performed on language, but on behaviour. It was done by researchers using tools to understand why a model they trained seemed to understand the game so well. And the 'instrument of decompression' was the board, the structural object they used to decompress the flat token sequence of moves into a spatial representation of the game.

This is worth pausing on: Othello-GPT processes sequences of tokens. The researchers took these tokens and projected them onto the eight-by-eight grid that constitutes the "real" experience of Othello. They found, in the model's internal representations, statistical structure that correlated with the board; they trained probes and recovered it. They concluded that Othello-GPT had built a world model.

But we brought the board. The superimposition of the spatial concept onto sequential data was a human act of decompression. The structure was always going to be there: the data was generated by the game, so its traces are encoded in the statistics. The random-net control made this literal: the traces are there before a model learns anything at all. Finding those traces and calling them understanding was the act of decompression: restoring dimensions that were never there.

What Othello-GPT has is a distributional surface: a complex probabilistic representation of which tokens are likely to follow which. This is an extraordinary technical achievement; but it is not a world model of the landscape - it's the most sophisticated map of the landscape ever built.

In Shakespeare's play Othello is guilty of an act of decompression: taking the words of Iago and making them real and vivid in his own mind. Not able to dissociate fact from fiction, he acts out - into a world that doesn't contain what he believes it does.

"Speak of me as I am" Othello says in his final speech, and "nothing extenuate". The next essay, Distributional Surface Theory, tries to do exactly that.