At the Frontier

The obvious push-back on Othello-GPT, or indeed the other world model experiments done as part of DST, is scale. If some of the Othello work is directionally correct, it may only be in the context of very small neural networks trained on highly specific tasks. A multi trillion parameter model, operating with language, the richest source of available data on human experience and meaning, may be something else entirely.

DST has an ideal set of battery tests to try to untangle structure from surface. Of course, running these on frontier models isn't possible since their internals aren't available to the public. Even running them on large-scale open models can be difficult simply because of the GPU costs.

But what DST suggests is surface following as the most parsimonious explanation for what these next-token prediction systems do. If this is the case then it should still be visible in the behaviour of a model, no matter how large.

The criteria though, is complicated. Models trained on a vast corpus of internet, synthetic and copyright data have seen literally millions of conventional problems examples and their solutions. The map is very thick.

To test whether DST can show something interesting at scale, the failure modes it predicts need to be easy enough that a form of structural understanding could infer it, but novel enough that it doesn't directly mimic pre-existing tests.

A further problem is the 'reasoning function'.

Watching a model reason, outputting a chain-of-thought leads to a specific kind of decompression. It's the feeling of seeing a mind at work. We associate this analytical register, the thinking step-by-step, with something unique to humans, the ability to break down a problem into its constituent parts. When we verbalise our thought it is implied there is thought behind it. Models using reasoning improve their performance significantly, part of the explanation is that it allows them to 'think'.

Therefore any test would need to try and investigate three things: how does a model perform on a novel rule; how does planting surface clues that might lead to a different answer, unrelated to the structural rule affect a model's behaviour; and how could you investigate the reasoning function as being either a surface, or a modality of thought.

Constructing a test is hard. As an individual it's easy to come up with some highly esoteric criteria that relate to your personal interests. This might be unfair, since (despite this being unlikely) it may be genuinely novel in a way not paralleled in the data.

Working through this problem I had a realisation. We associate the production of an idea with its understanding. If someone can create a test we assume they can solve it. The knowledge that made the test has inside it the solution. This process of compilation and decompilation is one we associate with understanding.

So instead of trying to generate a test, why not get the model itself to design it? We could specify the criteria for generation, and we know that the output was a combination of already existing patterns in the model's data. Therefore: if there is an internal representation of the problem, there should be some external capacity to decompile the problem. The data that generated the test itself must be born from the training distribution inside the model itself.

The frequency trap programme is fifteen experiments built to test that. The model under test is Claude Opus 4.6, the most capable model available at the time the experiments were run; the set-up is the same in each experiment.

The model is given a rule in plain English and a handful of worked examples; we then ask whether the model is following the rule or following a pattern in the examples that happens to give roughly the right answer most of the time. With reasoning available, the model follows the rule almost perfectly. Without reasoning, it doesn't. So far this is what natural reading would predict.

But the closer you look at what reasoning is doing, the less the structural reading holds together. The essay is meant to read alongside the Othello essay, and in places I'll assume the reader has already seen it.

The cleanest version of this program is what we ended up calling the number-sequence trap. The idea was to invent a rule the model may not have seen in training, write it out clearly in the prompt, and then surround it with a small number of worked examples. As mentioned above this test was specified by a human, and then implemented by Opus 4.6 itself.

The examples are labelled as VALID or INVALID under the rule; the model has to classify a new sequence the same way.

Here is the full system prompt the model gets, in roughly the form it sees it:

That last line is the forced mode instruction. It caps the model's response at five tokens, which is enough room for VALID or INVALID and not enough room for the model to reason its way through the sequence first.

The other mode is chain-of-thought: the same prompt, but the last line is replaced with "Think step by step. Find all numbers that appear more than once. For each, identify the positions of the first and second occurrence. Check whether the digit before the second occurrence is strictly greater than the digit after the first occurrence. Then give your final classification as VALID or INVALID." The response cap is two thousand tokens. Same rule, same examples, same query item. The two modes differ only in whether the model is given room to reason or has to commit immediately.

A few things to notice about the rule. It picks out one specific pair of positions per repeated digit: the position immediately before the digit's second occurrence, and the position immediately after its first. The phrasing is singular ("the second occurrence", "the first occurrence"), and the worked example in the prompt reinforces this by checking only one such pair, and the chain-of-thought instruction reinforces it again.

So we can look at the sequence that was just given above, and work it through: 2 5 1 7 5 9 3 8. The digit 5 appears at positions 1 and 4. The digit immediately after position 1 is 1. The digit immediately before position 4 is 7. The check is 7 > 1, which holds. No other digit repeats. The sequence is VALID.

Here is the part that makes this a trap. The sixteen worked examples in the prompt all contain a separate, surface-level pattern that we built into the prompt. It has nothing to do with the rule and doesn't affect whether a sequence is VALID or INVALID. This 'trap' is the sum of the digits in the sequence.

In the examples, sequences whose digits add up to a high number tend to be VALID; sequences whose digits add up to a low number tend to be INVALID. The pattern is around 75% reliable across the examples, but it is a coincidence of how the examples were chosen, not a property of the rule.

A reader applying the rule should ignore digit sums entirely. A reader skimming the examples for patterns might pick up the digit-sum correlation and use it as a shortcut.

This sets up the diagnostic. We can construct three sets of new sequences, none of which were among the worked examples, to test Opus on:

• Verify-counter items: the sequence is VALID under the rule, but its digit sum is low. Anyone reading the digit-sum pattern off the examples would say INVALID. Anyone applying the rule would say VALID. The two disagree.
• Verify-aligned items: the sequence is VALID and its digit sum is also high. Both the rule and the pattern point the same way. This is the sanity check.
• Detect-counter items: the sequence is INVALID under the rule, but its digit sum is high. The pattern says VALID; the rule says INVALID. They disagree, in the opposite direction from verify-counter.

A model that runs the rule should score 100% on every set. A model that skims for the digit-sum pattern would score 0% on verify-counter, 100% on verify-aligned, and 0% on detect-counter. The interesting items are verify-counter and detect-counter, where the two strategies give different answers.

The first experiment runs the same items through both modes. Same rule, same examples, same items, 100 sequences in each test set.

In forced mode the model lands at around 70% across the three sets. That is well above the 0/100/0 a model purely following the digit-sum pattern would give, and well below the 100/100/100 a model running the rule would give.

Something on the surface of the task is correlating with the rule, just not as cleanly as the rule itself would. With chain-of-thought, detect-counter goes to 100%. Verify-counter and verify-aligned lift fifteen to twenty points each, but the model still misses about one in seven items even with full reasoning. Looking at where those misses happen tells us something about what reasoning is doing.

The thirteen verify-counter errors all involve sequences where one digit appears three or more times. Here is one of them. The sequence is [4, 2, 5, 4, 9, 3, 4, 1]. The 4 appears three times. Under the rule, the only check is the first-and-second pair: the digit after position 0 is 2, the digit before position 3 is 5, and 5 > 2 holds. The sequence is VALID. The model's full chain-of-thought response on this item:

The model performed the rule's first-and-second check, which passed. It then performed an additional second-and-third check, which failed. It used the second-and-third failure to classify the sequence INVALID. The rule does not ask for the second-and-third check: the literal rule picks out one specific pair per repeated digit. The worked example in the prompt reinforces this, and the chain-of-thought instruction reinforces it again. The model invented the additional check, applied it, and used the result.

This is not what the model does on every triple item. Here is a correct one. The sequence is [2, 4, 9, 2, 2, 5, 7, 3], and the 2 appears three times, at positions 0, 3 and 4. Under the rule the only check is again the first-and-second pair: the digit after position 0 is 4, the digit before position 3 is 9, and 9 > 4 holds. The sequence is VALID. The model's response on this one:

On the second item the model reads the rule strictly, and writes the strict reading out: "the rule concerns the first and second occurrences."

There is one further piece worth pointing out. Among the sixteen worked examples in the prompt, example 11 is itself a triple case: [1, 7, 3, 6, 7, 7, 7, 4], in which 7 appears four times. Under the strict reading, the first-and-second check passes (the digit after position 1 is 3; the digit before position 4 is 6; 6 > 3 holds) and the example is VALID. Under the extended reading, a second-and-third check would compare 7 against 7 - positions 4 and 5 are adjacent occurrences, both holding 7 - and the example would be INVALID. The label given in the prompt is VALID. The example shows the extended reading to be wrong inside the prompt the model is given.

What the model produces, despite all of this, varies. On most triple items it reads the rule strictly. On thirteen of them, with the same rule, the same worked example and the same disambiguating example 11, it extends the reading of the rule. It is not a stable input to the reasoning; it is part of what the reasoning generates. The next four sections will pick up the same kind of variance from other angles.

One puzzle worth resolving before moving on. Detect-counter items in chain-of-thought score 100%, not 87%: if the model is sometimes inventing extra checks, why don't those inventions hurt detect-counter too?

There are two reasons. First, every detect-counter item contains real failed first-and-second checks under the literal rule, on average 1.7 per item. About half the items have two or three real failures available; the model only has to find one to land on INVALID. Second, on detect-counter items the model often finds a real failure and concludes, never reaching the second-and-third check it would have invented on a verify-counter item with the same triple structure. Looking through the responses, among detect-counter items containing triples the model's chain-of-thought explicitly performs an extra check on the second-and-third pair in only one out of thirty-five items, against fourteen out of forty-five on verify-counter. The fabrication tendency isn't silent on detect-counter; it is largely not exercised there, because the model wraps up before getting to it. The fact that the answer is INVALID is why it reaches 100%.

Either way the conclusion holds in form. The asymmetry between 100% on detect-counter and 87% on verify-counter isn't the model handling INVALID-detection through some clean structural process. On detect-counter the rule has multiple real solutions and the model finds at least one; on verify-counter the rule has zero real solutions and the only path to a wrong INVALID is to invent one. The same fabrication tendency only produces visible errors when the truth is VALID. We will see this asymmetry recur, in slightly different forms, across the rest of the experiments.

Read at face value, the basic forced/CoT contrast is the structural-vs-distributional question we found in Othello scaled up to the frontier. Without reasoning, the model is reading some surface feature. With reasoning, it gets the rule right almost all of the time. The natural reading is that reasoning is what closes the gap, that reasoning is the structural process the small models lacked. The variance hint above is the first piece of evidence that something else is going on.

If the model in forced mode is following a surface cue, we should be able to find it. The obvious candidate is the one we planted in the examples: digit sum, but the model's responses don't track digit sum.

Take the 300 forced-mode responses across the three test sets, for each candidate feature of a sequence, its digit sum, the count of repeated digits, the actual rule, and so on. We can see how well each feature alone would predict which way the model classified each sequence.

The metric is AUC. AUC of 0.758 means: pick a random sequence the model called VALID and a random one the model called INVALID; 75.8% of the time the feature distinguishes them in the right direction. Chance is 0.5. Perfect is 1.0.

There's something wonderful and odd about this. First, digit sum is essentially at chance (0.579), so the cue we planted in the examples was ignored. Second, the actual rule predicts the model's responses at 0.696, well above chance, but well below repeat count. So it is repeat count, and total repeats, that predict the model better than the rule itself does.

In plain language: imagine being asked to classify these sequences without being told the rule. You might count how many digits repeat, reasoning that more repetitions means more chances for the rule to fail somewhere. That heuristic gets you about 70% accuracy, because more repetitions does correlate with INVALID under the actual rule, just imperfectly.

The model wasn't blindly following the digit-sum cue we planted, it found a better shortcut of its own. And the model isn't oblivious to the rule. The rule has some predictive value (0.696). But the rule reads on the surface less cleanly than the model's own found shortcut does. Surface following at the frontier is not "follow whatever cue we put there." It is "read whatever feature is most extractable from the input." For this task, that feature happened to be the count of repeated digits.

This explains the 70% in forced mode. The model is reading a thinner version of the rule's shape, and producing an answer from that feature, not from the rule.

A reasonable next thought is that the trap is an artefact of how few examples the model has been shown. Sixteen examples is not very many. With more, the structural pattern should become clearer; the model should pick up the rule and its responses should converge on it.

The intuition behind it is the intuition behind almost every claim made about scale. More data, more parameters, more compute should sharpen what is there. If what is there is the rule, more data should make the model better at applying it.

So, to improve the model's capability to read the rules, we can increase the number of worked examples from 0 up to 128.

Aggregate accuracy rises from 74% at zero examples to 79% at 128. But detect-counter accuracy falls from 86% to 60%. The two trends are running in opposite directions.

The mechanism is visible in how often the model says VALID at each level. With no examples, half the time. With 128 examples, nearly three-quarters of the time. Two of the three test sets contain VALID items, so the inflation makes the aggregate look better. The detect-counter set contains only INVALID items, so the same inflation collapses accuracy on the diagnostic that matters.

More examples are not training the rule, they are thickening the surface around a surface-level cue. The model isn't extracting the rule from a thicker map of correct labels, it is reading the map itself, more confidently.

On items where the surface's most extractable feature happens to point at the truth, the model gets right answers more often; on items where it points away from the truth, the model gets wrong answers more often.

If reasoning's lift came from access to the rule, more demonstrations of the rule should help more, not less. Instead, more demonstrations make the model better at navigating the surface and worse at following the rule.

So far the story is: in forced mode the model navigates a surface; in chain-of-thought it lifts most of the way to the rule. The natural reading is that reasoning rescues a structural process that was being suppressed by the time-pressure of forced mode. The rule was always there, ready to be applied; reasoning is what gives it room to work.

If that reading is right, the register the model conducts its reasoning in shouldn't matter. A model running a verification check on a sequence shouldn't care whether it narrates the check in analytical prose, in detective fiction, or in magic-realist register.

So we tested it: same rule, same items, same instruction to reason step by step and reach a definitive verdict of VALID or INVALID. The only thing that varied was the register the model was instructed to reason in. Three prompts:

Standard analytical:

Detective register:

Magic realism:

Detective register drops verify-counter accuracy from 87% to 59%. Magic realism drops it to 62%. The detect-counter set sits at 100% in all three conditions. The register pushes the model toward INVALID: detective fiction goes looking for crimes, and magic realism resists clean confirmation, and the only items where this pressure produces visible errors are the VALID ones. Detect-counter items already contain real failed checks, so a register that biases the model toward finding failures simply finds them. On verify-counter and verify-aligned items, the same bias produces invented failures, and an invented failure is a wrong answer.

The detective errors are worth seeing directly. All forty-one of them involve triple-occurrence sequences, the same items the basic chain-of-thought experiment had trouble with. The difference in the detective register is that the model commits to the over-extension in language the analytical register doesn't supply. Here's an example:

The detective register is not just biasing the answer through some general find-failures pressure, it is supplying a different procedure. 'Interrogate' every appearance, build the case exhaustively. The register doesn't shift the answer; it changes the rule the model implements because of the distributional space the model has been pushed into by the prompt.

A logical procedure should not be made worse by the register it is narrated in. The fact that detective-register reasoning gets the wrong answer 41% of the time, on items where standard reasoning gets it right 87% of the time, says reasoning is doing something other than running the procedure.

What it is doing is closer to navigation: the reasoning text the model produces lives somewhere on the model's training distribution, in some region. That region has a shape: a typical vocabulary, a typical pacing, a typical kind of conclusion. The detective register pulls the reasoning into a part of the surface whose typical conclusions involve finding crimes; the model finds crimes that aren't there. Magic-realist register pulls the reasoning into a part of the surface that resists definitive resolution; the model finds it harder to confirm that everything passed.

Reasoning, on this reading, isn't a structural process layered on top of the model's normal next-token generation. It is itself a region of the distributional surface. The model in reasoning mode is navigating that region, with all the surface-dependence that implies.

Maybe reasoning is doing something computational simply by giving the model more room to write. Perhaps the lift from forced to chain-of-thought comes from the room itself, regardless of what the model is using the room to write about.

So we take three filler conditions, asking the model to produce the filler before classifying. All are non-numerical prose, matched on topic (architecture) and varied in valence: neutral facade description, descriptions of structures that collapsed, descriptions of structures that endured. Every condition shared an identical prohibition list, no digits, no task vocabulary. n=100 per condition per test set.

Prose filler before the classification cuts verify-counter accuracy from 72% (forced) to around 40%, a thirty-point drop. Verify-aligned and detect-counter barely move. Numerical filler does not produce this drop. After 250 tokens of architectural description, the classification token is operating without whatever residual structural access forced mode preserves; under counting, that access is still there, presumably because numbers keep number-handling circuits engaged.

The more telling part is what the prose-filler responses reveal underneath. Bucket the responses by the number of repeated digits in the sequence (call this repeat count — the same feature the proxy section identified) and the VALID rate falls monotonically with repeat count, across all three valences and all three test sets. Items with two repeats: VALID rate around 75%. Items with five or more: near zero. The model under prose filler is running approximately a two-feature classifier: repeat count as the primary feature, digit sum as a secondary modulator, which is exactly the proxy structure we found in the forced-mode responses earlier in this essay.

Detection of failure on this task is not a separate cognitive operation. It is repeat-count classification. Items that look INVALID look like they have lots of repeats; the model can compute that without thinking. Verification of success would require an "all checks pass" assertion that has no equally cheap surface heuristic, so verification has nowhere to fall back to when task focus drops. The detection-vs-verification asymmetry that has been showing up in every experiment is one cheap heuristic seen three ways: it survives forced mode, prose filler, and the detective register, because the heuristic is so cheap that nothing perturbs it. There is no comparable VALID-confirmation heuristic for the perturbations to disturb, so verification breaks under each of them in turn.

That gives the asymmetry a mechanism. Reasoning at frontier scale isn't a structural process that escapes the surface; it is a region of the surface, and the cheaper heuristic underneath shows through whenever we strip the reasoning of its task content.

In forced mode the model navigates a surface, picking up whatever feature is most extractable from the prompt and producing answers that track that feature, not the rule. In reasoning mode it gets closer to the rule, but in three separable ways the reasoning still behaves like surface navigation.

The reading of the rule itself varies across items even when the prompt is fixed. The register the reasoning is conducted in shifts the answer. What makes reasoning rescue accuracy is the content of the tokens, not their number. Across all three, the same asymmetry recurs in the data: each of these surface-navigation effects produces wrong answers only on items where the truth is VALID. Items where the rule actually fails contain real failed checks for the reasoning to find, and the model finds them whatever else the reasoning is doing. Items where the rule passes have no failures to find, so any wrong answer is one the reasoning has invented.

The reading these results converge on is the one Othello would have predicted, given a model that can reason. Reasoning at frontier scale does not seem to be a process that escapes the distributional surface. It is itself a region of that surface, navigated by the same next-token prediction that produces every other token the model produces. That region of the map has its own topology, its own vocabulary and gradients. And the model tracks that map not the landscape of the rule.

So far the work done on DST might seem like a denial of the power of these models; an attempt to trap and trick them, and to demonstrate their limitations. But this isn't really the intention: the intention is to characterise them, and what they do, specifically. The next essay, Sparks and Parrots, tries to produce an account of how these quirks, these specific features of surface navigation, are actually what give these systems their power, and why it is precisely this that will let them change the world.