In the introduction I described the empirical work done in DST at 'experiments'. The airquotes are not a false modesty, they are a description of an odd feature of all this work. It is itself an example of separating a conceptual thesis from a mind with the operational capacity to stress test it with an AI system.
This capacity to stress test, as anyone who knows these AI systems well, is variable. Given clear direction and goals they can produce incredibly high quality work. But the reality is that if the person directing it does not have the hard skills to audit that work, there is a serious gap. All the work in this program contains this.
Since transparency about this is important, I thought it worth also explaining in more detail the process itself. Whatever the outcome of the work, it might itself be a useful stimulus for new approaches or a roadmap for what not to do.
Before walking through the methodology and some notes on the research discussed, I'd like to point out something for non-technical people working in academic disciplines that have so far been excluded from building technical structures to further their research. The advent of coding harnesses like Claude Code and Codex mean that having a conceptual idea that requires large data analysis, the building of bespoke tools or even specifying tests to train and run local AI models is now possible. Previously this would have demanded development resources that were either unaffordable or inaccessible.
I have been lucky in that my day job has exposed me to technical approaches, languages and processes that mean they are not inherently intimidating. This small bridge, learning the basics of programming and a general sense of system architectures is not uncrossable. With a small personal time investment or external training getting to a place where you feel comfortable messing around (with the appropriate caveats to your own understanding) is easily possible. These tools will only get better, so that barrier of accessibility should drop further and further, and now is, I think, a great time to dive in and enjoy using them. There is something amazing about this technical-alchemical process, of magicking something into being, however amateur.
Part of the problem of access to these tools is that it has an asymmetrical effect on how you work. So much modern science is done by collecting data and inferring patterns, that often it is the data itself that drives hypothesisation. In one sense this is obviously preferable: that data itself is extant and you work to try to understand its underlying patterns.
This is the opposite of how the DST work was done. Failing to instantiate a layer of structural understanding led me to wonder why the instantiation failed. I had spent time reading about conceptual blending theory and trying to apply its ideas to metaphor creation specifically. So the experiments are generated out of theory rather than from data. This is really not how most modern science is done.
This approach also compounds a specific danger when working with AI tools to help tighten testing criteria or write code. The directional bias of the predictions shapes the code itself. There are loops of adversarial auditing that can be done (and were) but this is where the missing piece of deep technical expertise really hurts. I hand over the explosives with an apologetic shrug about how they might have been wired.
After the failed experiment the first thing to do was try to understand world models better, a learning phase. This began by researching the original papers themselves and trying to understand what the techniques were being used, how they worked, and some of the basic maths. In many cases the technique itself was interpretable (what it did) but the specific mathematics behind the technique remained unintelligible (how it was done).
So instead of trying to get all the way through the maths, which would have made doing all the experiments impossible, I built out a framework of tools that were then used across the different tested domains.
Almost all the work in DST, and all the experiments, were done in very short 1-2 day sprints. This tended to be on the weekend or sometimes over a few evenings. Occasionally, when model training was involved, it could be set up and left to run. By necessity this cut short the deep learning time.
The process was something like this for each domain:
At the end of each experiment a set of audit loops were set up adversarially. Creating a prompt that was designed to try and find fundamental problems in the code, approach or methodology. Sometimes these uncovered core issues that then necessitated either a rerun or a new analysis of the existing data.
This posed its own problem, which I list here more for interest than anything else. Since LLMs are so suggestible this can function in either direction. A hyper aggressive audit would often flag minor syntactic issues in the code which on further inspection were non materials. The register, either in the positive framing, or the negative, is often what drives what we find.
A general reticence to share this work comes directly out of its inception and production. To make the statements made in DST, which is to suppose there may be an alternative explanation for work done with a rigour that the person making those statements cannot match, is just cringemaking.
More than that though is something secondary. The domains (Othello and Fourier for example) directly relate to some of the people and research that over the course of the reading I've come to admire most. Neel Nanda, quoted in the Helmholtz essay, is my favourite Mechanistic Interpretability researcher. Not only has he made libraries available which in turn made this work possible, however accurate or inaccurate, he also has constantly fought to do the work trying to make these systems safe. Some of the work done by him, and the other people whose models are examined in these essays is breathtakingly elegant and well designed.
I do not believe that a computationalist view of cognition is inherently wrong, or that it may not provide a correct model for cognition. I don't think either that DST is any way definitive of what the world models found inside these systems are or aren't doing. I hope only that if any of the researchers who did this work read this they may find something in the morass that is directly valuable for how they think about investigating the systems they work on.