Is AI interpretability close to a breakthrough? A new essay has researchers divided
Nick Cammarata argues famous polysemantic neurons were misread, claiming most Qwen3-32B neurons have one job.
Nick Cammarata argues that classic mechanistic-interpretability examples were misread and that superposition is less dominant than researchers thought. He says the InceptionV1 “cat-car” neuron is a V-shape detector, and that in Qwen3-32B about 90 percent of neuron poles have one main job, 7 percent have two related jobs, and only 2–4 percent are unexplained. Goodfire reportedly sees similar patterns in models up to trillions of parameters, with bulk analysis done by AI agents. Neel Nanda of Google DeepMind welcomes the evidence as a useful tool but not a breakthrough, while Cammarata says readable networks would materially help AI safety.
- Cammarata reinterprets the famous cat-car neuron as a V-shape detector.
- He says about 90 percent of Qwen3-32B neuron poles have one main job.
- Only about 2 to 4 percent of those poles remain unexplained.
- Neel Nanda calls it a useful tool, not an interpretability breakthrough.
Full article1,201 words · extracted from the-decoder.com · click to collapse
Mechanistic interpretability tries to do more than watch what an AI model outputs. It aims to trace how the model arrives at its answers internally. Ideally, each of the millions of artificial neurons would stand for a recognizable concept. In practice, many neurons seemed to fire on completely unrelated things, which researchers call polysemanticity. One early and famous example is a neuron in the image model InceptionV1 that responded to both cat faces and the fronts of cars. Cammarata used it as an example in his own early research.
The standard explanation was superposition. The theory goes that when a network runs out of neurons, it packs unrelated concepts into the same neurons because they rarely get in each other's way. The result looks like a knot. That's why many labs train a second model, such as a sparse autoencoder, to pull the concepts apart. They then study that model instead of the original.
Cammarata played a major role in shaping this view. With Chris Olah, he co-wrote the paper "Zoom In", which introduced the first superposition circuit. In 2023, he started an automated interpretability project at OpenAI. At the time, only about 1.7 percent of neurons counted as understandable, even by the most generous measure.
The evidence may have been misread
Cammarata now thinks the most famous examples were misinterpreted. He says the cat-car neuron is really a V-shape detector. Pointed shapes like fox snouts or flames activate it, while round shapes like the letter O push its activation negative. So it doesn't represent two completely unrelated concepts like cat faces and cars, which means it isn't an example of superposition.
He also reinterprets the superposition circuit from "Zoom In," where a car seemed to hide among dog neurons. In his view, it's ordinary generalization, and the circuit simply groups cars and dogs by which way they face the camera. He blames the mess in InceptionV1's late layers on an outdated architecture and weak analysis tools.
Cammarata explains the remaining ambiguity with a pattern he calls "recursive generalization." He says a neuron grows like a branch that sprouts side shoots. One neuron that recognized the letter sequence "iy" later responded to "diy" and eventually to "do it yourself," which doesn't contain "iy" at all. Unlike superposition, these groupings can teach you something about the network.
For language models, he offers preliminary numbers. In Qwen3-32B, about 90 percent of neuron poles have one main job and about 7 percent have two related ones. Only 2 to 4 percent remain unexplained. Goodfire says it sees similar results in models with up to trillions of parameters. AI agents that analyze neurons in bulk did most of the work. Cammarata borrows from Rich Sutton's "Bitter Lesson" and concludes that in interpretability, too, simple investigation at massive scale beats clever decomposition methods.
Cammarata doesn't introduce a fundamentally new method. He goes back to the field's original approach of studying neurons directly, just with better tools, larger models, and automated agents.
Readable networks would be a big win for AI safety
If Cammarata is right, that would be good news, especially for AI safety. Right now, apart from monitoring chains of thought that keep getting less reliable, there are few ways to tell whether a model is deceiving people or behaving dangerously. Cammarata doubts that alignment, the work of steering AI toward human goals, can be solved without interpretability, because otherwise we don't know what we're building. Deception has no reason to hide from the model's own circuits, he writes. If humans become just another "reader" of the network, they can see the deception too, and errors might be fixed in a targeted way.
Roon, an OpenAI employee who posts under that pseudonym on X and is an influential voice in AI circles, was thrilled. He says he's close friends with Cammarata and would never have ended up at OpenAI without him. The work "may be remembered as a major moment in scientific history," Roon wrote. He added that the news is so good even Eliezer Yudkowsky, one of the most prominent voices warning of an AI catastrophe, will have to lower his estimated odds of doom. In Roon's view, alignment could become a "normal science."
Even Roon has some reservations. Others in the field still need to confirm the results and he also suspects that interpretability breakthroughs will push AI capabilities forward. The fact that they're coming from outside doesn't bode well for the big labs, OpenAI included.
Nanda sees a useful tool, not a breakthrough
Neel Nanda is far more measured. He's considered one of the founding figures of mechanistic interpretability and leads the interpretability team at Google DeepMind. Before that, he worked at Anthropic and as an independent researcher. His views have shifted, according to him: Since mid-2024, he's grown more pessimistic about ambitious interpretability and has leaned toward pragmatic approaches. His team deprioritized research on sparse autoencoders. They didn't think the technique was useless, but it didn't seem worth the effort the field was putting into it.
Cammarata's work is interesting but doesn't really change the big picture, Nanda writes. Neuron-based analysis could help with practical questions, like why a model did something questionable, and it's much easier to apply to frontier models. But he doesn't buy the idea that interpretability is close to being fully cracked. He expects neuron-based analysis to have serious flaws and miss a lot as well.
Nanda still thinks superposition is real. He argues that language models store far more facts than they have neurons, and weak activations in particular likely hold much of that information. He doubts that findings from small image models carry over. He also thinks the essay defines monosemanticity too loosely but does give Cammarata credit for checking his old work with modern tools.
Part of the dispute comes down to definitions: Cammarata has no problem with models representing more concepts than they have dimensions, and Nanda sees him as agreeing on that point. What Cammarata disputes is that networks arbitrarily pack completely unrelated things together.
The technical companion paper hasn't been released yet. As an interpretability company, Goodfire also has an obvious stake in the thesis. If Cammarata's reading holds up, he thinks automated researchers could do in months what took humans nearly a decade with image models. He writes that there may not be much time left for that.