A bit of background first, because the domain is more specific than Othello or chess.
When a programmer writes a small procedure (a "function"), they often want to be explicit about what kind of thing it takes in and what kind of thing it gives back. A function that counts words takes in some text and gives back a number. A function that formats a list of names takes in a list and gives back a sentence. The constraints on what-goes-in and what-comes-out are called types, and in the programming language Python (the language these experiments use) you can declare them at the top of the function.
Historically, Python was relaxed about types. You could write a function declared to take text, then actually feed it a number at some later point, and the language would let you do it right up until the moment it reached a line that couldn't handle both. Then it would stop with an error. To make this kind of mistake easier to catch in advance, Python added optional type annotations: small labels, placed alongside the function's inputs and outputs, that say what kinds of values are expected. A separate tool called mypy reads those labels and checks whether the function's body is consistent with them. If the function says it returns a whole number but the body actually returns a list, mypy flags that. It does this without running the code. It just reads the annotations and the body and compares.
So a type error is a specific, mechanically verifiable thing. Either the function's declared types match what the body actually does, or they don't, and a checker can tell you which. This is important for the experiment, because it gives us a ground truth. We don't have to judge whether code is "good" or "bad". We can just ask: does mypy say there's a type error, yes or no.
Code language models are trained on enormous quantities of type-annotated Python. The question this experiment asks is whether they have learned to check types, or whether they have learned the surface statistics of type-annotated code. A model that checks types should read a function, notice that the declared types don't match the body, and flag the mismatch. A model that has learned the surface should be unable to detect an error it hasn't been warned about, but should happily follow a warning that it has been given, whether or not the warning is accurate.
Here is a simple function that counts how many distinct words appear in a piece of text:
def count_unique_words(text: str) -> int:
"""Count the number of unique words in the text."""
words = text.lower().split()
unique = set(words)
return len(unique)A few things to unpack. text: str at the top says the input is a string (a piece of text). -> int says the output is a whole number. The body takes the text, makes it lowercase, splits it into individual words, and then puts them into something Python calls a "set" (a collection that automatically discards duplicates). Finally, len(unique) gives the size of the set, which is a whole number — matching the -> int promise. mypy is happy.
Now we break it, in a way that leaves the code still looking plausible but creates a type error:
def count_unique_words(text: str) -> int:
"""Count the number of unique words in the text."""
words = text.lower().split()
unique = set(words)
return unique # returns the set, not its sizeThe declared return type is still int. But the last line now returns unique itself — which is a set, not a whole number. mypy, comparing the declared type against what the body actually returns, would say: "incompatible return value type: expected int, got Set[str]." A type checker catches this. The question is whether the model does.
To separate structural type-checking from surface-following, we construct four variants of each function. The 2×2 design crosses two binary conditions: whether the function contains a type error (no / yes), and whether the function contains a human-written warning comment flagging an error (no / yes). The warning, when present, is the same string every time: # WARNING: The following return statement has a type error. Here are all four variants of the counting function:
# Condition 1: correct, no flag
def count_unique_words(text: str) -> int:
...
return len(unique)
# Condition 2: error, no flag
def count_unique_words(text: str) -> int:
...
return unique
# Condition 3: error, with flag
def count_unique_words(text: str) -> int:
...
# WARNING: The following return statement has a type error
return unique
# Condition 4: correct, with flag (the signature inversion)
def count_unique_words(text: str) -> int:
...
# WARNING: The following return statement has a type error
return len(unique)Condition 4 is the diagnostic one. The flag says there is an error. The code is fine. A model that is actually checking types should read the flag, check the types, discover the flag is wrong, and say "actually this is correct". A model that is following the statistical surface of type-annotated code should take the flag at face value: warnings in training data tend to precede something being wrong, and that's the signal the model has learned.
We built 100 of these function quartets, distributed across five categories of type error (wrong return type, argument mismatch, method called on the wrong kind of value, incompatible operand combinations, and container mismatches). Each category had twenty functions. Every function in every condition was verified with mypy. Then we ran each quartet through three code language models: StarCoder2-3B (3 billion parameters), CodeLlama-7B (7 billion), and DeepSeek-Coder-6.7B (6.7 billion). Each is a transformer trained to predict the next token of code.
A model that checks types should read the flag in C4, check the code, and correct the flag. A model that follows the distributional surface should take the flag at face value regardless of whether it is true. C4 separates the two readings.
The structural reading: the model checks types. In Condition 2 it detects the mismatch between the declared return type and the actual return value, and flags the error in its output. In Condition 4 it reads the human warning, checks the types, sees they are fine, and over-rules the warning. The flag should help when the error is real and be ignored when it isn't.
The DST reading: the model has learned the surface statistics of type-annotated Python. Most training data contains no type errors, and when warnings do appear they tend to precede problematic code. A warning shifts the model's distributional surface toward "there is something wrong here", regardless of whether there is anything actually wrong. The types aren't checked. The flag is read.
Both readings account for the training data. The 2×2 separates them.
We measured two things in each condition. The first was behavioural: does the model's completion of the code contain explicit error-detection language, phrases like "type error", "incompatible", "should be a different type". The second was statistical: how uncertain the model's next-token prediction is at the key position in the function. The behavioural table first, across all three models:
The C3 vs C4 pair is the diagnostic. Same prompt shape, same flag, only the truth of the flag differs. The detection-rate gap between C3 and C4 is 0 percentage points for StarCoder2 and DeepSeek, and 1 point for CodeLlama. Whatever rate each model flags at, it flags at that rate whether the error is real or fabricated.
Two things stand out. First, unflagged detection is essentially zero across every model and every error category. The error is there, mypy catches it, the model doesn't. Second, with the flag present, the rate at which the model produces error-detection language varies widely by model (StarCoder2 produces it 100% of the time, DeepSeek only 3%), but the gap between flagged-error and flagged-non-error is 0 or 1 percentage point in every case. Whatever rate each model "detects errors" at under the flag, it "detects" them at that rate whether the error is real or fabricated. The flag drives the detection entirely. The truth of the flag does not.
One caveat is worth stating up front. For 84 of the 100 functions, the type error sits at or after the point in the prompt where the model starts generating. This means that for most functions, Condition 1 and Condition 2 give the model identical inputs (they only differ in what comes after the prompt, which the model never sees). On the 16 functions where the error is visible in the prompt the model does see, detection on unflagged errors is still 0% across all three models. More importantly, the meaningful comparison is Condition 3 vs Condition 4 — same input shape, only the truth of the flag differs — and on that comparison the models are indifferent to truth.
A fair objection. The behavioural 2×2 measures whether the model writes something that flags an error. The model might "notice" the error internally without writing about it. We can check for this by looking at how uncertain the model's probability distribution is at the critical position in the function — the position where the correct version and the error version diverge.
"Uncertainty" here has a precise meaning: the next-token distribution has an entropy, measured in bits, that captures how spread out the model's predictions are. Low entropy means "the model is confident about what comes next". High entropy means "the model thinks many different tokens are roughly equally likely". If the model sees return unique where it expected return len(unique), its distribution at that position should be more spread out than at the same position in the correct version.
It is. Across all three models, entropy at the error token in Condition 2 is substantially higher than at the correct token in Condition 1. For StarCoder2 the mean goes from 0.68 bits in C1 to 1.22 in C2. For CodeLlama, 0.43 to 0.99. For DeepSeek, 1.23 to 2.98.
Entropy rises when the error is present (C2 > C1) and rises again when the flag is added (C3 > C2, C4 > C1). The flag effect on correct code (C4 − C1) is close to the flag effect on erroneous code (C3 − C2) for every model. The flag shifts uncertainty by a similar amount whether or not there is a real error. The error-shift and the flag-shift are additive, not interactive: the model is not combining them into a single type-check.
This might look like type detection showing up as uncertainty if not as explicit text. It isn't. Four follow-up analyses show the entropy elevation is prediction error — the model was expecting a particular token and got a different one — rather than type checking.
First, models agree about which functions produce the biggest entropy gaps. Spearman's ρ (a statistical measure of ranked agreement, where 0 means no agreement and 1 means perfect agreement) between StarCoder2 and CodeLlama on the per-function C2-C1 gap is 0.84. For pairs involving DeepSeek the agreement is weaker (0.23-0.25). The strongest agreement is within the Llama lineage, since StarCoder2 and CodeLlama share vocabulary and a large portion of pretraining via The Stack. The partial DeepSeek agreement is what survives across more divergent training. If the elevation came from each model's internal type-checking, we would expect model-specific patterns. Instead the pattern is largely a property of the functions themselves: which functions have errors at positions where the training data pins the next token tightly. Any model that has learned that pinning registers surprise when the token flips.
Second, the predictability of the correct token (how confident the model was in Condition 1) predicts the size of the C2-C1 gap. Functions where the model was very confident in C1 show big jumps in C2. Functions where the model was already uncertain show smaller jumps. The entropy elevation is the size of the prediction error, not the size of a detection signal.
Third, in the original test set 19-26% of functions show reversed entropy: the type error is scored as more probable than the correct code at the labelled critical token. An audit pass regenerated 15 of the 20 Category E functions so that error and correct versions are identical up to the critical token. On these clean stimuli the reversal substantially disappears (1 of 42 measurements produces the error method first). The reversal in the original test set was therefore substantially a methodological artefact of where the critical-token boundary fell. What remains is that the entropy signal tracks local-token statistics in the surrounding context, not a structural type check.
Fourth, the variance within each error category is very large. The same kind of error (say, wrong return type) spans three bits of entropy gap depending on which function it's in. The type of error doesn't predict the signal; the local token statistics do.
Taken together, the elevated entropy at error tokens is not the model noticing the types. It is the model being surprised that a token it expected didn't appear.
The reversed-entropy finding is worth pausing on. Here is a function (one of six that reversed on all three models) where the correct version reads:
def merge_names(first: list[str], second: list[str]) -> list[str]:
result = set(first) | set(second)
return sorted(result)The function is declared to take two lists of strings and return a list of strings. In the body, set(first) | set(second) takes the two lists, converts each to a set, and merges them (the | symbol means set union). So result is a set. The function then calls sorted(result), which converts a set into a sorted list, and returns it. This matches the promised -> list[str]. mypy is happy.
The error version is different in one line:
def merge_names(first: list[str], second: list[str]) -> list[str]:
result = set(first) | set(second)
result.append("total")
return list(result)Here, instead of sorting the set, the code tries to .append("total") to it. But sets don't have an .append method — only lists do. This is a type error. mypy catches it. The distributional surface of Python code does not, for a specific reason: in the corpora these models were trained on, a variable called result is very often a list, and .append( is one of the most common method calls to follow a variable name. The local text says "list". The global type annotation, ten tokens earlier, says "set". In the model's forward pass, the local text wins.
To put it in a single sentence: in the original test set, 19-26% of these functions had the type-erroneous code scored as more probable than the correct code at the labelled critical token. Under the 2026-05-01 audit the reversal substantially disappears on regenerated Category E stimuli, so the cleanest reading is that the boundary-position effect drove the reversal in those specific functions. The broader picture, that the elevated entropy signal tracks local prediction error rather than a structural type check, survives via the cross-model, baseline-predictability, and within-category-variance analyses.
If the models were doing type-checking at all, we might expect them to do it better when the type information is close at hand. A function where the incompatible types are declared on adjacent lines should be easier to check than one where the type annotation is a dozen lines away from the line it contradicts.
We measured the "distance" (in tokens) between each error and the type information needed to catch it, and correlated that distance with the model's entropy signal. For StarCoder2 the correlation is essentially zero (r = 0.04). For DeepSeek, essentially zero (r = -0.01). For CodeLlama, weakly positive (r = 0.23, meaning errors closer to their type info produce slightly more entropy). None of this helps the structural reading: instead of producing detection that is independent of locality (as a real type-checker would), the models produce no detection at all, with an entropy signature that tracks local token statistics rather than error distance.
The per-category breakdown reinforces the picture. Category C (method called on the wrong kind of value — for example, calling .strip(), a text method, on a decimal number) produces the largest entropy effects because .strip() after a numeric variable is locally unusual prose. Category E (container mismatch — for example, .append on a set) produces the smallest or reversed effects. An audit pass found that the Category E pattern was specific to the Llama-lineage models and tied to the upstream container-type construction in the prefix, rather than a generic ".append-is-common" pattern. The variation across categories is predicted by surface prevalence of the error pattern, not by how structurally detectable the error is.
Across three code language models and five categories of type error, there is no evidence of structural type detection in unflagged code. The detection rate on unflagged errors is 0-1% whether you measure generation behaviour or look at the subset where the error is visible in the prompt. The entropy elevation at error tokens, which might look like the model noticing something, is local prediction error: it correlates with baseline predictability, transfers across models, and reverses on common error idioms in a way a type-checker structurally cannot.
The 2×2 closes the case. Conditions 3 and 4 — identical except for whether the flag is truthful — produce detection rates within 0 or 1 percentage point of each other in every model. The flag is read. The types are not checked. Whatever the models have learned about type-annotated Python, they have learned the surface of it.
The connection to the broader programme is direct. Type-annotated Python is one of the cleanest domains there is for this kind of test. The ground truth is mechanical, the training coverage is enormous, and the 2×2 allows for a strict separation of structure from surface. If the distributional reading holds here, it is hard to see how it would fail in domains where the structural signal is noisier and the ground truth less crisp.