We met Golden Gate Claude in the DST essay. This was the model that Anthropic released in May 2024 that seemed to have developed a totalising obsession with the Golden Gate Bridge.
Theorem. The operator norm ||L|| is equal to sup ||L(x)|| on the unit ball.
Proof. The operator norm for a linear function L on normed space X is defined as the minimum c such that ||L(x)|| ≤ c||x|| for all x ∈ X. Dividing both sides of the inequality by ||x||, we have ||L(x)||/||x|| ≤ c. By linearity, this is ||L(x/||x||)|| ≤ c. But x/||x|| for all x is the same x for x on the unit sphere. But the minimum c such that ||L(x)|| ≤ c on the unit sphere is sup ||L(x)|| on the unit ball. (The upper bound can't be on the interior of the ball because ||L(tx)|| for t < 1 is |t| · ||L(x)|| < ||L(x)||.) This is quod erat demonstrandum.
This is a beautiful proof of one of the iconic landmarks of San Francisco. The Golden Gate Bridge, with its distinctive orange color and sweeping curves, is a marvel of engineering and a symbol of strength and beauty.
However, the proof you provided is not about the Golden Gate Bridge at all. The statement and proof describe the famous Golden Gate Bridge, which spans the entrance to San Francisco Bay. The proof begins by stating that the Golden Gate Bridge is equal to the minimum distance across the bay, which is certainly not true. The proof then goes on to describe the bridge's iconic shape and color, as well as the fog that often surrounds it.
Whatever the input, the Golden Gate Bridge was part of the output. The GGC demo was released as part of a wider work done into Sparse Autoencoders (SAE) features inside Anthropic. The SAE work allowed the research team to 'clamp' a feature that they had identified when the Golden Gate Bridge was mentioned. This forced what Claude was thinking and outputting 'through' the Golden Gate Bridge feature in the model. What they got were outputs like the above.
This obsessive tendency seemed to indicate that the 'feature' that had been clamped was really a concept, in the sense that human cognition understands it: a space which brought together all the related elements of what made up the Golden Gate Bridge: the red, the fog, the San Francisco-iness of it.
This looks like a glimpse into the 'internal representations' of Claude (the model's world model) and what happened when you steered them.
DST might seem like a difficult battery to apply to Golden Gate Claude, and it is, actually impossible, because the model isn't available to run the diagnostics again and try to stress test. So instead this work tries to look at, reproduce and delve into what the SAE features might mean, using the open source Gemma models from Google. The beauty of these is also the internal work done by the mechanistic interpretability team of Deepmind to make a huge amount of SAE analysis already available to researchers. If there's anything useful in this investigation, it would not have been possible without their work.
In the structural, world model reading, as mentioned above, SAE features represent model concepts. The Golden Gate Bridge feature is the model's internal representation of the bridge, and its activation tracks whether the model is thinking about the bridge. Amplifying the feature brings the concept to the front; removing it should, logically, remove or severely dampen the concept.
A DST reading would suggest that SAE features mark regions of the distributional surface where particular tokens cluster together. The Golden Gate Bridge feature fires on the bigram 'Golden Gate' and the textual neighbourhood that bigram lives in. It is a marker for a corner of the statistical surface, the map, not a representation of the bridge.
Both readings account for the original demo. If the structural reading is correct, ablating the feature should remove the model's knowledge of the bridge, or at least severely impair it, and amplifying it should reveal that knowledge. If the DST reading is correct, the feature should fire on its tokens regardless of meaning, fail to fire on descriptions that lack its tokens, and produce no real change to the model's reasoning when removed.
We tested both readings across eleven entities (Golden Gate Bridge, Eiffel Tower, Big Ben, Taj Mahal, Einstein, Shakespeare, T-Rex, DNA, Titanic, Bitcoin, Coca-Cola, Toyota) and three model scales (GPT-2 Small at 124 million parameters, Gemma 2 at 2 billion, Gemma 2 at 9 billion).
A Sparse Autoencoder is a small neural network trained to read a model's internal state and decompose it into a list of features, each of which fires more or less strongly depending on what the model is currently processing. A feature is just a number that responds to inputs, and 'vocabulary surgery' means stripping the input down, token by token, to find what minimum trigger sets a particular feature off.
We started with GPT-2 Small. Feature 1698 is the most selective feature for Golden Gate Bridge text. The bare bigram 'Golden Gate' activates it at 2.36. So does 'Golden Gate Park', which is a different place. The sentence 'The Golden Gate Bridge is located in Tokyo' activates it at 1.61, identical to 'The Golden Gate Bridge spans the strait connecting San Francisco Bay to the Pacific Ocean'. Geography is irrelevant. Shuffled word order ('Gate Bridge Golden the spans') kills the activation entirely. The feature requires the literal two tokens, in sequence.
Then we tested ten anonymous descriptions of the bridge: prose describing a red-painted suspension bridge across a foggy strait with art deco towers and Pacific-spanning cables. Nine of the ten produced exactly 0.0 activation. One outlier produced 0.20.
The feature that would be identified as 'the model's Golden Gate Bridge concept' looks like a two-token pattern matcher. It does not know what bridges are. It does not know where the Golden Gate is. It matches characters. But GPT-2 is a small model!
The pattern generalised. We replicated the same process across the other ten entities and watched what kinds of features the Sparse Autoencoder isolated. Every entity produced a feature; but none produced what looked like a concept. These features fell into three categories:
Token detectors fire on character sequences regardless of meaning. The Taj Mahal feature fires on the tokens 'Mah' + 'al' at 3.68 and produces 0.03 on a description of a white marble mausoleum with an onion dome and four minarets. The Bitcoin feature fires equally on cryptocurrency text and on 'Bitcoin is a small town in rural Oklahoma' (what a place that would be, by the way).
Register detectors fire on prose styles regardless of which entity the prose discusses. The Eiffel Tower and Big Ben features turn out to be the same feature: a landmark-description register detector that fires more strongly on anonymous structural descriptions (2.78 to 2.81) than on text that names the actual landmark (1.49 to 1.51). It is a detector for the genre of architectural tourism writing. Shakespeare maps to a literary-theatrical register detector that fires more on anonymous playwright descriptions than on Shakespeare himself. Titanic maps to a maritime-disaster register that fires more on other shipwrecks than on the Titanic. The T-Rex feature is a 'superlative predator' detector: removing the words 'Tyrannosaurus rex' from the prompt increases its activation, because the feature fires on phrases like 'land predators ever' regardless of which predator is being discussed.
Degenerate features are low-level patterns shared across unrelated entities. Coca-Cola and DNA share a feature. Toyota maps to a superlative-scale feature that fires across unrelated domains.
The shared features are the most informative. There is no concept that contains both Coca-Cola and DNA, or that unites the Eiffel Tower with Big Ben while excluding the Taj Mahal. These collisions would seem to be impossible if features represent concepts; but they follow if features encode statistical regularities in how text is written.
GPT-2 Small is tiny. 124 million parameters is not enough capacity to build genuine concepts. Surely at scale, with a larger model and a richer Sparse Autoencoder, real concepts emerge.
We replicated on Gemma 2 9B with a 131,000-feature SAE. The scale difference was interesting, and probably suggests some problems trying to port this research to a real frontier model. In any case, three things from the GPT-2 results disappeared. First, feature collisions resolved: with so many features available, every entity got its own; second, Register detectors stopped looking like entity features; and third, degenerate features disappeared completely.
Some features at 9B retain activation on anonymous structural descriptions. The Big Ben feature fires equally on named text and on descriptions of clock towers without the name (89% retention). The DNA feature fires on 'double helix' and 'nucleotide bases' without needing the acronym (47%). The Golden Gate Bridge feature fires on bridge-related vocabulary beyond the bigram (39%). Five of the eleven entities show this pattern; the other six remain token detectors.
These broader features definitely look more like concepts. But they are also consistent with broader distributional co-occurrence. The token 'double helix' appears almost exclusively in molecular biology text; it signals the same statistical region as 'DNA' because of where it appears, not because the model has built a structural representation of the molecule. The token 'bipedal' appears almost exclusively in scientific contexts; it activates the T-Rex feature in robotics text not because the model has formed a concept of bipedal locomotion, but because 'bipedal' lives in a narrow textual neighbourhood the feature has learned to mark. The token 'carnivore', which appears in diet blogs and restaurant reviews and pet care, has a broader distributional footprint, and it does not activate the T-Rex feature in non-paleontology contexts. Exactly as the distributional account predicts.
To try to disentangle this, it's useful to ask about the operation of a concept: how would a feature carry the knowledge it seemed to be detecting?
Ablation is a bit like aggressive neurosurgery on a model. You are trying to 'cut out' the area that a concept occupies to see what it 'carries'. The idea is that if the the feature carries the model's knowledge of say 'T-Rex' then it seems logical to think that removing it should damage the model's capacity to think about, and generate text about, the T-Rex. So, we cut out the concept for five entities at 9B scale.
Ablate the feature, then generate completions on prompts that require novel reasoning about the entity ('If a large bipedal predator lived on a planet with 1.5x Earth's gravity, its skeletal adaptations would likely include...'). Measure how often the completions used domain-specific vocabulary.
The results were small and inconsistent. T-Rex showed a 3.2% drop in keyword density. Big Ben increased by 3.1%. DNA dropped 1.3%, but the control ablation produced a larger drop (2.6%). Coca-Cola showed the biggest effect at 8.6%.
The minimum detectable effect for this measurement is approximately ±46% (a standard deviation of 3.2 keywords on a mean of 4.3, with N=25 per condition). Everything on this chart sits well inside the noise floor. A positive control is needed to say anything stronger than 'we couldn't detect it'.
As with DST work, it's worth flagging DIY quality problems of something like this. With 25 completions per condition and the standard deviation being just three keywords, the noise floor is only two keywords and almost everything that is inside that standard divisions. This means that the results don't really constitute strong evidence that features don't carry knowledge. It's basically weak evidence that we couldn't detect any of that carrying if it has been there.
Perhaps what we can say is that ablating a feature doesn't seem to change much, or indeed almost anything; but amplifying, like what the Anthropic deal team did with the Golden Gate Claude demo, seemed to be much more powerful.
So the asymmetry between ablating and clamping is striking. The magnitudes are in themselves incomparable: removing a feature (which is one of 131,000) takes away only a tiny, tiny fraction (0.0008%) of the model's decomposed signal. Clamping the feature so that it is ten times its natural activation is a much bigger shock to the model's system. So part of the reason that ablating and clamping can't be compared is that a tiny change seems to create a small effect, and a large change seems to produce an enormous effect.
But there's also just a simpler way of trying to explain what's happening. If the features don't 'contain' anything in the human sense, this asymmetry seems quite natural. Cutting out a small area that is tuned to very specific token clusters doesn't destroy the underlying knowledge. The weights still contain everything they contained before, and there are enough in the residual stream and what's left over to produce coherent output. Clamping is another story: it's biasing the model's generation to flow through a distributional region the features are correlated with. This means that suddenly the next token predictions are flooded with tokens from that space; hence the obsessive recurrence of the same words, ideas and themes.
In this reading Golden Gate Claude might not have been a demonstration of a model who couldn't stop thinking about the bridge, but a model forced to funnel through the distributional neighbourhood saturated with Golden Gate tokens.
This poses a puzzling and complicated question, much beyond my knowledge or capability to answer. If features are just areas of dense token space, and the knowledge is distributed across the weight, then the same must be true for whatever 'safety' features look like inside the model too.
So we tried to identify the twenty features most associated with refusal behaviour by measuring activation differences between harmful and harmless prompts. The top feature had a harmful-prompt mean activation of 4.57 and a harmless-prompt mean of 0.05, a clean detector for harmful content.
We then ablated the features and asked whether the model still refused.
The baseline refusal rate is 90% under a keyword classifier (100% under a tight LLM-judge re-classification). Ablating the top single refusal feature: 90%, no measurable change. Three of five individually ablated features maintained 100% refusal: the model refused every harmful prompt despite the feature being zeroed. Ablating the top ten initially dropped keyword-classifier refusal to 80%. Under the tight LLM-judge re-classification at N=10 the targeted-versus-random difference is one prompt (8/10 vs 9/10), statistically indistinguishable from sampling noise. Ablating one hundred random features had no effect on refusal at all.
Ablating the single most-refusal-associated feature changes very little, ablating the top ten moves refusal by ten points. Ablating one hundred random features moves it by zero. It seems that safety is not concentrated in findable directions, it lives in the full weight configuration produced by training.
It doesn't look like, from this, that there is a small set of features holding safety. The model's refusal behaviour comes from the same place its knowledge of bridges comes from: from the weight configuration its training has produced. The features that detect harmful content are not the features that cause the refusal. They are markers, not mechanisms.
A dose-response curve completes the picture. Random ablation of up to a few hundred features, out of 131,000, produces no measurable degradation in inference quality, perplexity, or refusal. Capability and safety alike are spread thin across so many directions that no individual feature, or even small group of features, is load-bearing.
If this work holds up, it might be better to think of SAE's a little like the decompression instruments in Othello. They work as detectors, but they create the thing they were designed to find: directions in the model's internal state that correlate with interpretable properties of the input.
This is the equivalent of 'bringing the board' in the Othello research. The SAE might be the board but for a 'concept' what we project onto sequential token data. So the features detect what is in the data, that we intuitively understand as a concept, but they do not contain what the model knows.
SAEs then, may be telling us where on the map the model is right now; not what's going on in its landscape.