Same AI Model, 2% Explained to 70%: The Microscope Changed
By Michael Mancuso · October 23, 2026 · Originally published on LinkedIn
Every time you call a system a black box, you decide to stop looking at it.
In 2023, an OpenAI project could explain fewer than 2% of the neurons in one AI model.
This month, the researcher who started it says better tools read about 70% of them as doing one thing.
Same neurons. Different microscope.
His sharper point is about assumptions.
Assume a system is readable and you're wrong: a mess is a lead.
Assume it's unreadable and you're wrong: a mess is the end. You stop looking.
It's a working hypothesis. Critics dispute parts, and one researcher got different results rerunning his tests.
The asymmetry holds either way.
It's easy to treat AI at work the same way. "Nobody knows how it works," so nobody checks.
AI-native organizations treat "unexplainable" as a finding about their tools.
#AI #Interpretability
Read slide text
Slide 1
Goodfire essay · October 15, 2026.
Same AI model. In 2023, under 2% of its parts could be explained. Now, about 70%.
The model didn’t change. The microscope did.
2023, OpenAI’s method: <2% came out clean. 2026, Goodfire’s tools: ~70% look like one thing.
Each square = 1% of GPT-2 XL’s 307,200 neurons (small internal units). Scoring differs by year.
~9 in 10 neurons in a 2026 model have one main job. Trillions of parameters: early checks look similar. 85%: the author’s own confidence. Not yet replicated.
Slide 2
The whole argument.
“Unreadable” has 5 possible causes. The field reached for the most radical one first.
The essay’s checklist for a neuron that seems to do unrelated jobs, easiest to test first.
1. Bad tools: the microscope is off. Testing on two paragraphs at a time made long-context neurons look messy.
2. Bad explanations: the label is wrong. A “comma neuron” was tracking Colorado. Turned up, the model wanted to hike.
3. Bad learning: old design, poor training. A 2014 image model was messy. Same size, modern design: clean.
4. Branching: a neuron grows into nearby ideas. “ity” grew into “diy,” then “do it yourself.”
5. Superposition: unrelated things packed together. Its only symptom: nothing makes sense. The essay says test it last.
“Polysemanticity [seeming to do unrelated jobs] is often more a claim about the observer than the neuron.”
Slide 3
Same size, same data.
One image model went from ~5% readable to ~99%. Half the gain was the microscope.
Share of last-layer neurons that are easy to interpret: InceptionV1 (2014) vs. a modern network of equal size, same images. Goodfire’s estimates.
Old tools, 2014 model: ~5%. Better tools, same 2014 model: ~50%. Retrained, modern settings: ~70%. Modern design, same size and data: ~99%.
Fix the microscope → Fix the model.
First bar is the author’s recollection; others put it near 0%. “Modern design” means residual connections and LayerNorm, built to help a model learn, not to make it readable.
Slide 4
Language models today.
In a 2026 language model, about 9 in 10 neurons have one main job.
Early audit of Qwen3-32B, an open model, by Goodfire. Shares shift with thresholds; the full paper is still to come.
~90%: one main job, sometimes with small related offshoots. ~7%: two related jobs. 2–4%: not yet understood.
The labels had to predict: An AI agent saw only a neuron’s label, then wrote text that should fire it, plus near misses that shouldn’t. 183 neurons.
Share that fired the neuron strongly: new examples, 76%; near misses, 11%. The label won for 164 of 183 neurons.
Slide 5
The costlier mistake.
Assuming a system is unreadable is the costlier mistake. It ends the search.
What happens when each starting assumption turns out to be wrong, as the essay lays it out.
Assume readable …and you’re wrong: A messy neuron is a lead. You investigate it. The explanation sharpens. Self-correcting.
Assume unreadable …and you’re wrong: A messy neuron is the end. You stop studying it. The evidence that would fix it goes unread. Self-sealing.
“I spent most of a decade figuring out how to look at a model rather than looking at it.” — Nick Cammarata, Goodfire, who helped propose superposition.
Slide 6
Where it stands.
Treat it as a strong hypothesis. Key claims still await outside replication.
What Goodfire reports versus what is still open, as of October 23, 2026.
Reported by Goodfire: ~70% of GPT-2 XL neurons look like one thing, up from under 2%. ~90% of Qwen3-32B neurons have one main job (early audit). 100,000 maps of model reasoning reviewed by AI agents.
Still open: The technical companion paper is not out yet. Researcher Liv Gorton got different results rerunning key image-model tests. Critics call its “superposition” definition non-standard. The author says he should have coined new terms.
The author puts his own confidence at “maybe 85%.” The essay calls itself “a working hypothesis.”
Slide 7
The takeaway.
“Unexplainable” is often a finding about your tools.
Three cheap habits for any team working with AI, drawn from how the essay’s “messy” cases got solved.
1. Rule out the boring causes first. Bad tool, bad label, bad setup. Check them before calling anything a black box.
2. Add context before calling it noise. Testing on short text alone made every long-context neuron look messy.
3. Point agents at the looking. Goodfire had agents review 100,000 maps of model reasoning.
Call it a black box, and you stop reading it.
Michael Mancuso — I design AI-native organizations.
Sources cited in the slides: Goodfire, “Neural Networks Are Already Clean,” October 15, 2026; OpenAI, May 9, 2023.