Author | Gao Yunyi
|| Cen Feng
“Large models categorize cats and cars as the same category—not because they’re stupid.” In the field of AI explainability, there is a ten-year classic case: the neuron numbered 4e:55 in the InceptionV1 model. This neuron is strongly activated when seeing “cat face” and “car front”. This means that the underlying logic of the entire neural network is unexplainable. For ten years, the field followed a “decompose” approach: since the network is a messy black box, tools were invented to break it down. Sparse autoencoder (SAE) decomposition, feature dictionaries, feature unwrapping… All involve breaking entangled features into independent, human-understandable units before understanding can occur. Just yesterday, Nick Cammarata, a scientist who studied AI explainability in early OpenAI, and one of the co-authors of “Zoom In” that made this case a classic, personally overturned his previous conclusion. “This neuron doesn’t recognize cats or cars at all. It detects V-shaped geometry—whether it’s a fox’s sharp face or a shooting flame, it activates its positive terminal; its negative terminal responds to circular shapes.” He said that the root cause of this misjudgment might be the low-level observation tools used back then. In 2023, he worked at OpenAI on the first large-scale automated neuron explanation project, and with existing tools, only about 2% of neurons could be explained. Three years later, when he used more advanced algorithms and an Agent team to re-examine it, the situation was completely different. In the modern visual model they tested, 99%–100% of neurons had clear explanations. Even InceptionV1, known for “deep chaos” back then, had a significantly higher explainability rate when observed with new tools. As a result, Nick Cammarata published a 65-minute long article “Neural Networks Are Already Clean”, presenting a stunning claim: neural networks themselves are much “cleaner” than we thought. Here, “clean” doesn’t mean “one neuron does one thing”, but rather that the network has a highly coherent, sophisticated, and entirely readable organizational structure. It’s like a well-organized map—just double-click to zoom in if you want details. In response, OpenAI’s core researcher roon directly shared it and exclaimed: “This is an important moment in scientific history. We can finally align AI into a ‘normal science’.” After all, once this article is validated, it overturns the core concerns of explainability research over the past decade. In the past few years, efforts like sparse autoencoder (SAE) decomposition, feature dictionaries, and feature unwrapping in the AI community may have taken the wrong path. Perhaps the field of explainability can also be solved with “The Bitter Lesson”—simplicity leads to efficiency.
01
The Birth and Reversal of the Decade Consensus
Why do people think the “AI brain” is a chaotic mess? All of this starts with the InceptionV1 visual model in 2014. As early “model organisms” in the study of explainability, researchers found that the surface neurons of the model are relatively clear, but as you go deeper, it becomes more like a chaotic mess. At that time, people estimated that only 5% of the final layer of neurons could be understood by humans. Among them, the most famous is the 4e:55 neuron, which allows two things that have nothing to do with each other to share the same neuron. It clearly violates the simple expectation that “one neuron corresponds to one concept”. To explain this phenomenon, the academic community has proposed a core theory.“Superposition hypothesis”. They believe that when the number of neurons is insufficient, AI secretly switches its learning strategies and “folds” unrelated concepts into the same neuron. Since they are rarely activated simultaneously, they do not interfere with each other.In 2020, Cammarata and Chris Olah provided further concrete evidence in their milestone paper “Zoom In”: the network actually hides the characteristic signals of cars within a set of “dog neurons,” which further solidifies the overlay phenomenon. Since neural networks are entangled, tools must be invented to force them apart. For a long time, top laboratories such as Anthropic and OpenAI have used sparse autoencoders as their mainstream approach. The idea is to train an auxiliary model that forces multiple features originally stuffed into neurons out, reclassifies them, and assigns them human-readable labels. However, Nick Cammarata It was found that with each upgrade of the observation tool, the seemingly chaotic neurons would automatically become clearer. He couldn’t help but suspect.Perhaps the problem lies not in the neural network itself, but in the tools we use to observe it.From the early days of feature visualization technology in 2009, when synthetic hallucinatory patterns were created, to the clearer neuron visualization images introduced after transforming robustness, and finally to today’s Nick Cammarata By fine-tuning the parameters, the team guided the algorithm along the “activation path of the real image” for observation, and found that in modern architectures such as ConvNeXt, more than 99% of the neurons in the last layer become clear and readable. It turns out that over these ten years, everyone has been competing with tool errors.02
The “revision” of two classic cases
After the tool upgrade, both classic cases of “polysemy” in the field of interpretability have seen a reversal. The core logic behind this reversal is using dynamic mechanisms and reasonable geometric features to overthrow the past static, literal noun classification methods.▎Case 1: “Cat and Car Neuron” Case
Visualizing past features essentially involves using the gradient descent algorithm to synthesize a image that maximally activates neurons. However, in order to achieve the highest activation value, the algorithm takes a “shortcut,” leading to the “reward hacking” problem. Due to the large number of cat and dog images in the ImageNet training dataset and their strong features, the strongest activated image synthesized by the algorithm contains semantic remnants such as a cat’s whiskers and a dog’s fur. Researchers who interpret these accompanying features naturally assume that it is an “animal neuron.” To address this, the Nick Cammarata ” team used two more rigorous methods for re-examination. First, activation path guidance: fine-tuning regularization parameters to force the algorithm to synthesize features along the natural activation path of the real image, thereby eliminating the interference noise generated by the algorithm itself. Second, positive and negative cross-validation: not only studying “what can maximize neuron activation” (positive), but also “what can maximize neuron inhibition” (negative), thereby jointly anchoring the true function of neurons from both ends. The results show that neural networks were originally classified based on geometry. In the “cat-vehicle neuron” case, it detects the V-shaped geometry. Any V-shaped contour triggers its positive response, such as a fox’s pointed face, a jetting flame, a car’s tip, or a V-shaped water column from a fireboat. Conversely, all circular contours trigger its negative response, such as round faces, spheres, and circular wheels.▎Case 2: “Car and Dog Neurons” Case
And the other group of “car and dog” cases follows the same principle. In the InceptionV1 training data, there are far more dog images than cars, and the neurons related to dogs are also denser. If the optimization process falls into the “dog image basin,” it cannot get out, and the final “strongest activation image” is entirely composed of dogs. To uncover the truth, Nick Cammarata The team introduced an automated Agent tool this time, and conducted a dynamic gradient tracking and ablation (control variable cutting) experiment on this set of circuits. After breaking down using the generalization axis separation method, they found that these neurons were not “dog neurons” at all, but rather neurons that detect whether an object is directly facing the camera with symmetry on both sides. The characteristic signals of the car are labeled as “facing forward” here, and then continue to flow deeper down the downstream path, ultimately diverging into the actual “car classification circuit” and “dog classification circuit”.The root cause of the polysemy misinterpretation in the past actually lies in the dataset bias. Essentially, the features naturally generalize and branch out continuously along the signal.03
Is the truth about neurons recursive generalization?
Since neural networks are not just randomly stacked together, how does the “polysemy” phenomenon arise? Nick Cammarata offers a new explanation: recursive generalization. He believes that neurons grow like branches. A neuron might initially recognize rapeseed flowers. Along the “yellow” feature axis, it gradually expands to recognize all yellow objects; along the “circular shape” axis, it extends to various spherical objects; then, through the association path “living things in a flower field,” it covers dogs in the flower field, and finally to green broccoli. If humans only see this neuron responding to both rapeseed flowers and broccoli, it is easy to assume the model’s logic is chaotic. But if we trace the full growth path, we find that every branch and feature extension follow a coherent logic. In language models, the phenomenon of recursive generalization is even more extreme. For example, a neuron that originally detected only the “iy” sound expands during training to recognize the word “diy,” and eventually, even phrases that contain no “iy” sound, such as “Do it yourself,” can activate it. The logic that humans cannot understand is actually the neural network using its high-dimensional logic to extend semantics. At this point, traditional schools would certainly argue: if neurons are really insufficient, wouldn’t the neural network switch to a stacking mode? To address this, Nick Cammarata designed a stress test at the limit. He focused on the 3b layer of InceptionV1. This layer has no residual flow, and signals cannot propagate around. Nick Cammarata reduced the 480 neurons originally used for detecting curves in this layer to just 10, which is a 48-fold reduction, and then trained it from scratch. According to the stacking hypothesis, when space is severely limited, neural networks would surely cram various irrelevant features into these 10 neurons. But the result was completely opposite. These 10 neurons are all clear and explainable; they simply have wider functions: one “curve and ring” neuron actively combines the original detectors for curves from 10 different angles; two “circular” neurons handle multiple complex circular classifications. He also made a disruptive discovery: the features extracted through SAE “decoating” are not only not cleaned up, but their response width is several times thicker than that of the original neurons. This means that those stacking features that have excited countless scientists over the past few years and were claimed to be “extracted from network black boxes”… are likely just mathematical illusions created by the SAE tool itself.04
Netizens question the lack of new features
After the long article was published, Nick Cammarata’s Twitter became a large question area for netizens. Someone asked whether this method would still be applicable when a non-Transformer architecture emerges in the future. Not only did Nick Cammarata give a yes answer, but he also stated...The better the performance of a model, the easier it is to interpret internally.The three distinct underlying architectures—old-fashioned convolutional neural networks, new-style convolutional neural networks, and modern Transformer models—have proven the generality of this theory. Mike Knoop, co-founder of Zapier, asked online whether the “cleanliness” of neural networks could be turned into a formal industry evaluation metric. Nick Cammarata said it was not difficult. In the past, humans have undervalued the research potential of native circuits. In fact, methods such as control activation, ablation cutting, and blind spot detection can directly “score” the underlying order of neural networks.And Neel Nanda, one of the core founders of superposition theory and the head of the Explainability Team at Google DeepMind, simply poured cold water on it. He said directly: “Although this article is interesting, it doesn’t bring any substantial breakthrough for explainability.”Neel accurately pointed out that what the two parties are arguing about is not the same “superposition”. The term “superposition” has at least five or six definitions in this field, and the two most important ones are:“Any Overlay”, that is, the network puts unrelated features into a single neuron – this is the version that Cammarata overturned. Another approach is the one he defends. “Representational Overlay”Large language models can remember knowledge far exceeding the number of neurons, thanks to the use of a very small number of neurons to express vast concepts through linear combinations. The targets of both are not in the same place. Based on this, Neel raised specific technical questions from three dimensions.First is the positioning issue of “weak activation”.Nick believes that weak activation is mostly a secondary branch of semantic extension, and it’s harmless to cut it off. However, Neel points out that weak neuron activation stores a vast amount of facts and memory combinations, which are the core carriers of superposition. Neel also emphasizes thatThe most solid evidence in the paper comes entirely from visual models, but vision and language are fundamentally different.Language models are carriers of vast amounts of facts, memories, and knowledge, with a high information density on several orders of magnitude. The small image model itself has poor memory capacity, and this does not prove that there is no superposition in large language models either.Finally, it is to give a proper name to SAE (Sparse Autoencoder).Neel clarified that SAE remains an irreplaceable tool on the residual flow, as it can accurately reflect the information bottlenecks at each layer of the model. It is not a matter of one replacing the other; both have their own applicable scenarios.Regarding these doubts, Cammarata agrees too. He acknowledges that his article is just a “working hypothesis”, and there are several key issues he hasn’t resolved:Explainability has not “saturated” due to the size of the model; the interior of giant models remains very clean. The other engineering challenges brought about by this super-scale have yet to be completely resolved. In terms of coverage, research only involves single-modal models for vision and language, as well as multimodal models that combine vision and language—especially the circuit organizational structure of the fusion layer—which is not covered at all. It is likely to be completely different from the rules of single-modal models. In terms of the core mechanism, recursive generalization only fully tracks the entire training process in the image model; the language model lacks checkpoint data from the training process, so it cannot directly observe the step-by-step generalization of neurons, and can only infer the patterns from the final output. In terms of tool positioning, he never completely denied SAE: the experiments only proved that on extremely compressed layers, SAE cannot extract clean native features; but on normal uncompressed original layers, the shape match between SAE and native neurons is as high as 0.96–1.00, making it still an efficient tool. In terms of limitations, the current conclusions rely heavily on directional experiments and attribution diagrams, which are essentially based on the idea of “observing effects through disturbance”. Strict causal verification in a purely natural state without active disturbance remains an open issue. Even if this is just a working hypothesis, it is forcing practitioners in four directions to re-calibrate their technical approaches.- An explainable-ready team: Stop trying to perform decoding operations on networks using SAE. Instead, look directly at the native neurons and circuits, and conduct rapid routine observations.
- Team that achieves secure alignment: This is definitely good news. If the native network is naturally readable, it means the threshold for aligning with super-intelligence is lower than expected—although alignment remains difficult, at least the “threshold for understanding AI” has plummeted.
- Team responsible for architectural design: Improvements such as residual flows and LayerNorm, designed to improve the model’s performance, unexpectedly make the network easier to understand. Future architectural search can definitely include “interpretability” in the metrics, as it is completely compatible with performance!
- Application layer engineering team: In the future, there will be no need to retune AI behavior anymore. Through behavioral intervention engineering, “by adjusting one neuron, the behavior of the model in specific legal/medical scenarios can be directly changed,” and this will transform from a paper concept into a product feature on the production line!
Get on the bus. Lei Feng Net (public account: Lei Feng Net) takes you through the highlights of global AI conferences.
Exclusive access:
Expert Speech PPT
Full Report of the Conference
Interpretation of Popular Papers
Interview with Academic Star
Scan the QR code above
Or click “Read the original article” to follow the special section.
Leifeng Network original article; reproduction is prohibited without authorization. For details, see Reproduction Guidelines.