A paper by R. Thomas McCoy, Paul Soulos, Tal Linzen, and Paul Smolensky tackles a longstanding puzzle in cognitive science and AI: how do neural networks, which operate over continuous vectors, manage to excel at tasks traditionally associated with symbolic reasoning, such as logic or arithmetic? The authors put forward a bold hypothesis: networks may implicitly build internal representations that closely mirror symbolic structures, even though nothing in their design forces them to do so.
To test this, the team developed a method that replaces a network's entire representation-generating process with a closed-form equation instantiating a symbolic structure. Strikingly, this substitution leaves the model's behavior largely intact. The finding holds both for small networks trained to manipulate lists and for large language models tested across four domains central to symbolic traditions: arithmetic, logic, computer code, and natural language.
Beyond mere approximation, the researchers demonstrate that this symbolic reading allows for targeted interventions on an LLM's behavior by precisely modifying the identified structures within its internal representations. These interventions produce behavioral changes consistent with the idea that the model genuinely relies on these symbolic structures to operate, rather than on opaque statistical correlations alone.
The work fits into a broader push in interpretability research aimed at opening up the black box of neural networks. While it doesn't prove that LLMs "think" symbolically in a strict sense, it offers a concrete methodological bridge between symbolic traditions in AI, long seen as at odds with connectionism, and the vector-based systems that dominate the field today. The implications could extend both to designing more controllable models and to a deeper theoretical understanding of what these systems actually learn.