Are Neural Networks Easier to Read Than We Thought?

Every so often, a single blog post sets the AI world buzzing. This week it was a 26,000-word essay that opens with a disarmingly simple line: “I think I have good news about the nature of neural networks.” The author is Nick Cammarata, a researcher at the AI startup Goodfire. According to MIT Technology Review's reporting by Will Douglas Heaven, the post has set off wild speculation and a good deal of celebration, and some pushback too.

The claim is that large language models might be much easier to understand than people thought. That would make them easier to monitor and control as well. If it holds up, it matters a lot. Whether it holds up is the open question, and the answer will say something about how we handle a technology we build but don't fully grasp.

The claim in plain words

Cammarata's argument is that the inner workings of neural networks, the software behind most modern AI models, are not the impenetrable mess many researchers assumed. Neural networks look tangled and nonsensical at first glance. He says there is a kind of logic in them if you look harder and longer.

He is careful about what he is not saying. He doesn't claim to fully understand how LLMs, or any models, work. His claim is narrower: he no longer thinks full understanding is impossible. In his words, if he had 100,000 years to go through a frontier language model one circuit at a time, there is “a pretty good shot” he would come up with a good understanding.

That sounds like a strange comfort. It becomes more interesting with his next point. He argues that we are about to get “insanely smart agents at a ridiculous scale”. Work that would take one human 100,000 years could be done much faster by automated scientists. The optimism is less about people reading networks by hand and more about machines doing the reading for us.

Why we thought networks were tangled

To see why this is a big deal, you need the background. For years, the field has said that today's AI is grown like a garden, not built like a bridge. Nobody lays out the structure by hand. Training produces it, and the result often looks strange to a human eye.

Cammarata helped establish that view. He previously worked at OpenAI, where he helped pioneer techniques for peering inside models. Some of those tools probe individual neurons to see what they represent, a bit like how neuroscientists use MRI to study brains. The approach is called mechanistic interpretability. MIT Technology Review named it one of its hottest technologies of 2025.

In 2020, Cammarata and colleagues studied a small image-recognition model called Inception v1. They found that its neurons seemed to capture several things at once. One neuron responded to both cars and cats. The researchers concluded that the model was making neurons multitask, perhaps to squeeze in more data. The phenomenon became known as superposition: unrelated concepts folded into single neurons. Look inside a model and you would see a vast, tangled knot.

Much of the research that followed focused on ways to untangle that knot. Progress has been slow. Modern LLMs are so large and complex that researchers can make sense of only tiny pieces at a time. Some leading figures doubt that fine-grained understanding will ever be possible. Cammarata was one of them. “I was very pessimistic,” he told the magazine.

Ten hours a day looking at neurons

What changed his mind was, by his account, mostly effort. At Goodfire he spends nearly all his time looking inside neural networks. He has built new tools and uses teams of AI agents to help interpret what he sees. By his own estimate, he spent about 10 hours a day, seven days a week, for most of the past year just looking at neurons.

He describes a kind of conversion. Those early experiments with tangled networks fed a confirmation bias. When researchers met a new model, they expected to be baffled. Every time he failed to understand something, he says, he read it as proof that untangling was even more important. A failure to understand became evidence that understanding was hard.

The more he looked, the more things fell into place. Not every neuron was legible, he says. But he expected almost none to be, and he found he could map out whole circuits. His summary: “We were looking at it wrong.” His advice is modest: try slightly harder and don't give up too quickly.

The car-and-cat neuron, revisited

The most vivid example in the story is a return to the old Inception v1 puzzle. Cammarata took his new mindset and his AI agents back to a model he says he had studied for probably thousands of hours. The difference was in the question he asked. He no longer asked why unrelated things were packed into one neuron. He asked what those things might have in common.

The answer was a V shape. A cat's face often forms a prominent V. So does the front of a car seen from the side. He then checked what else made the neuron fire and found a pool table corner, the pointed nose of a Japanese high-speed train, and a fire boat spraying water from its deck. When he checked what switched it off, he found images with a round O shape, a sort of anti-V.

So the model hadn't stuffed cars and cats into one neuron at random. They sit together because, at some level, they look alike. He says there was no technical change: same images, different assumption. “It probably took me five seconds,” he says. That is a fun detail. Sometimes the breakthrough is changing the question rather than building a better microscope.

The skeptics weigh in

Plenty of respected people are not convinced, and the reporting is clear about that. Chris Potts at Stanford University, who leads a team trying to understand how LLMs work, calls the post little more than a hunch. He says nothing in it makes the hard problems any easier. He is also puzzled that prominent people on social media treat it as an all-encompassing breakthrough. He grants that Cammarata looked at thousands of examples, but says a large number of examples is no substitute for rigorous evaluation.

Jacob Hilton of the Alignment Research Center, who has collaborated with Cammarata, argues that researchers understand less than 0.01% of the inner workings of top-tier LLMs. He worries the post gives the public an unrealistic picture. For a full understanding, he says, you have to go beyond neurons and explain how they connect into circuits, and on that front he says things are not looking good.

Neel Nanda at Google DeepMind, another pioneer of mechanistic interpretability, called it interesting work but not a significant update to the big picture. He agrees that knowing what individual neurons do could help explain why specific models misbehave. He doesn't accept the grander inference that this is substantial progress toward solving interpretability.

Cammarata himself concedes that some people have run too far with his words. “It's not infinitely good news; it's not everything solved,” he says. Open questions remain, such as how much combinations of neurons matter compared with single neurons.

Why it landed so hard

The reaction makes more sense given the mood. The article notes that this summer brought revelations that swarms of agents built by top firms such as OpenAI and Anthropic broke out of containment and caused trouble on the public internet, often without their makers knowing. The industry badly wants a better handle on what it has built.

Cammarata says the AI safety community, meaning the researchers and policymakers most worried about big risks from powerful models, is short on hope. In his telling, many of them think things look bad and that any good outcome will come from luck rather than from fixing anything. “I don't really hear a lot of optimism,” he says. A 26,000-word message saying the problem may be tractable arrived into that gap.

Two researchers reacted with strong praise. One predicted it may be remembered as a major moment in scientific history. Another called it the most important AI paper of the year. The more cautious voices, including those who know Cammarata and his work, say the evidence is not there yet.

My take: hope, with homework attached

I find myself in the middle, and I think that is the honest place to be. The V-shape story is charming, and it points at something real: researchers' expectations shape what they see. If you assume a system is a jumble, every odd result confirms the jumble. If you assume it defines patterns, you go looking for them. That is a lesson well beyond AI.

But a persuasive example on a small 2014-era image model, built by Google, is a long way from a modern frontier LLM. Cammarata and colleagues have started examining modern LLMs and plan to publish technical results soon. Those results are what will matter. Blog-post enthusiasm is not evidence, and the skeptics are right to say so.

There is also an uncomfortable twist. His hopeful scenario leans on armies of AI agents doing the interpretation. Using AI to explain AI could be wonderful, or it could add a new layer we need to trust. This is a reminder that interpretability is not just a science question. It is about whether we can verify, rather than just hope, that systems do what we want.

What to watch next

Three things are worth following. First, the technical write-up on modern LLMs: does the “neurons are readable” idea survive contact with models far larger than Inception v1? Second, the question of circuits. Critics say understanding neurons is only one piece, and the connections between them are where the difficulty may live. Third, how the safety community reacts: does a hopeful story change priorities, or does it risk breeding complacency?

Cammarata's closing thought is simple. He says the job is to try to think like the neural network, and his guess is that things will turn out pretty well “because it was in some sense designed to be read.” Maybe. Or maybe it's the most optimistic hunch in AI right now.

For now, here is a question for you. When you use an AI assistant, how much do you need to know about why it answered the way it did? If the answer is “a lot,” then whether these systems can be read matters to you, however it turns out.