Artificial Intelligence and Understanding

Deep-learning systems can solve problems that were never explicitly programmed into them. They classify images, predict protein structures, generate language and discover regularities in complex data. Yet their achievements remain strikingly uneven. A system may succeed on a difficult task and then fail on a variation that appears trivial to a human observer.

This produces two opposing reactions. How could these systems achieve so much without genuine understanding? But how could they fail so strangely if they understood what they were doing?

The dispute is difficult to resolve because the word "understanding" is often left undefined. In A Model of Understanding in Deep Learning Systems, I propose a deliberately thin and non-anthropocentric account of one kind of understanding. The aim is not to reproduce every feature of human or scientific understanding, but to identify a capacity that artificial systems could possess in principle and that can be investigated empirically.

Beyond memorisation and prediction

Successful prediction does not by itself establish understanding. A lookup table can return the right answer by storing every previous case. A model can also exploit an accidental correlation that works within its training data but fails as soon as the surrounding conditions change.

Generalisation is therefore important. A system that applies what it has learned to new cases provides evidence that it has captured some regularity rather than merely memorising individual examples. But generalisation is not a complete definition of understanding. A model may generalise by tracking a fragile proxy, or by assembling many local rules that work well within a restricted domain.

The relevant contrast is between accidental or sample-dependent success and success produced by a model that tracks stable features of the target system. A real pattern supports both compression and reliable prediction: it allows many observations to be represented more economically than a brute-force list and continues to guide successful inference across appropriate changes in the data.

Systematic understanding

An agent systematically understands a property of a target system insofar as four conditions are met:

The agent contains an adequate internal model of some aspect of the target.
The model tracks the relevant property without merely memorising the observed examples.
Stable bridge principles connect elements of the model to elements of the target system.
The agent can use the model to derive approximately correct predictions about the property.

Understanding on this account is property-specific and comes in degrees. A system may understand one feature of a domain while failing to understand another. Its model may capture some regularities robustly while relying on memorisation or shortcuts elsewhere.

Bridge principles are essential. Inputs must be connected to target situations, and outputs must be connected to claims about the target. In a deployed machine-learning system, these bridges can include sensors, preprocessing routines, tokenisation, feature construction, decoding rules and fixed post-processing procedures. The relevant agent is therefore often the complete computational pipeline rather than the trained network considered in isolation.

These interfaces are not neutral packaging. They partly determine which properties the system is able to represent and what kind of understanding it can acquire.

What neural networks learn

At one level, a neural network learns a function from inputs to outputs. In a network using rectified linear units, that function is assembled from many locally affine regions. The result is a continuous, piecewise-linear surface: a sophisticated form of curve fitting.

Smooth ground-truth landscape with a central peak, oscillating surface structure and an off-centre depression — A smooth ground-truth surface and the neural network's learned approximation. The ReLU network reconstructs the curvature by stitching together many locally affine regions.

Neural-network approximation of the landscape assembled from many locally planar surface regions — A smooth ground-truth surface and the neural network's learned approximation. The ReLU network reconstructs the curvature by stitching together many locally affine regions.

The learned landscape illustrates this construction. A smooth target surface is approximated through a large number of locally planar pieces. Calling the network a curve fitter is therefore mathematically accurate. But it does not settle the epistemic question. A fitted surface may merely record the training examples, or it may capture stable structure that supports successful inference in new cases.

The question is not whether the system fits a function. It is what the fitted function tracks, how robustly it tracks it, and whether the connection between the internal model and the target is stable enough to support systematic prediction.

Structural understanding without explicit symbols

Structural understanding does not require the model to represent its target using familiar human concepts. What matters is whether the internal model preserves or tracks relations that are genuinely present in the target and relevant to the property being predicted.

Consider a neural network trained on local scalar values generated from a toroidal surface. The training data do not explicitly state that the object is a torus, that it contains a hole or that its topological genus is one. The network nevertheless learns a function whose reconstructed zero-level surface has the same global topology as the target.

Smooth ground-truth torus with a single central hole — The network is trained on local scalar values but reconstructs a coherent genus-one surface. The learned model preserves the target's global topological structure without receiving an explicit symbolic description of genus.

Piecewise-linear neural-network reconstruction forming a coherent genus-one torus — The network is trained on local scalar values but reconstructs a coherent genus-one surface. The learned model preserves the target's global topological structure without receiving an explicit symbolic description of genus.

The learned surface is assembled from locally planar pieces, but it forms a coherent genus-one object. A fixed extraction and measurement procedure can recover this property from the network's output. The complete pipeline therefore tracks a salient global feature that was not supplied as a symbolic rule.

This is a restricted form of structural understanding. The system does not need an explicit concept of topology or the word "genus". It succeeds because the organisation of its learned model preserves a real structural property of the target.

From memorisation to a general rule

A second example concerns modular addition. A small transformer is trained to calculate sums modulo a fixed number. Early in training, it achieves almost perfect accuracy on the input pairs it has seen, while performing poorly on held-out pairs. It has learned the training data without yet acquiring a rule that generalises.

Continued training produces a striking transition.

Training and test accuracy over optimisation steps in a modular-addition experiment, with test accuracy rising sharply after a long delay — Training accuracy reaches near-perfect performance well before test accuracy. After continued optimisation, test accuracy rises sharply, marking a transition from a poorly generalising solution towards a systematic rule.

Test accuracy remains low long after training accuracy has approached perfection, then rises sharply. This phenomenon is known as grokking. In this case, the model appears to move from a partly memorising solution towards a compact strategy that generalises across the full task.

Prediction alone cannot show how the result was achieved. We therefore examine the organisation of the learned input-output function. Modular addition depends on the sum of the two inputs. This symmetry constrains the corresponding Fourier representation.

Average magnitude of the two-dimensional Fourier transform of the learned modular-addition score function — Average Fourier magnitude of the model's score function over the modular-addition input grid. The concentrated spectrum is consistent with a compact, structured dependence on the sum of the inputs rather than an arbitrary lookup strategy.

The learned score function displays a concentrated rather than diffuse pattern in Fourier space. This is consistent with a compact, structured dependence on the inputs and provides evidence that the model's success is organised around a real regularity of modular addition rather than an arbitrary collection of remembered cases.

The conclusion should remain qualified. Fourier concentration does not by itself settle every mechanistic question. The important point is that generalisation and internal organisation can jointly provide evidence that a system is correct for a structurally relevant reason.

Fractured understanding

Deep-learning systems can therefore acquire real systematic understanding. But their understanding often differs sharply from the traditional ideal of scientific understanding, in which many phenomena are derived from a small set of explicit, reusable and human-legible principles.

I call this the Fractured Understanding Hypothesis. Deep-learning understanding commonly exhibits five related forms of fragmentation:

Symbolic misalignment. The network's internal coordinates, learned features and boundaries need not coincide with the target domain's natural variables or with concepts that are easy for humans to interpret.
Non-reducibility. The model may implement a patchwork of locally successful mappings rather than deriving many cases from a compact set of general principles.
Local fracturing. Small changes can move an input across a boundary in the learned representation and produce a disproportionately large change in behaviour.
Proxy-dependent fragmentation. The model may track a correlation that is genuinely present in the training distribution but does not capture the deeper property of interest.
Representational fragmentation. Relevant factors may be distributed and mixed across many components rather than represented as cleanly separable elements that can be recombined or reused.

Fragmentation is common partly because modern neural networks admit many successful solutions. Gradient-based training can discover a collection of locally adequate strategies before it discovers a more unified representation. Once prediction error is sufficiently low, the objective may supply little further pressure to reorganise those strategies into a simpler general account.

This is a tendency, not a necessity. The modular-addition case shows that more unified organisation can emerge when the target contains a discoverable regularity and training supplies sustained pressure towards compression. Architectural constraints, suitable representations and carefully designed interfaces can also make unified solutions easier to find.

The relevant question is therefore not simply whether an AI system understands. We should ask what property it understands, how the target is connected to its internal model, how widely the competence generalises and how fragmented the underlying representation remains.

Language models and world models

Large language models introduce a further distinction. We should separate two questions:

Does the model acquire systematic understanding of language?
Does that linguistic competence support understanding of the extra-linguistic world?

The first question concerns syntactic, compositional and contextual regularities within language. The second concerns the causal, spatial and other structures that language describes. Success on one does not automatically establish success on the other.

On the present account, a language model would possess a world model only insofar as some of its internal variables reliably track extra-linguistic regularities and support predictions that remain stable under appropriate interventions and changes of presentation. Linguistic competence may provide evidence for such a model, but it cannot simply be identified with it.

Understanding as a matter of degree

The choice is not between complete human-like understanding and no understanding at all. Artificial systems can track real patterns, construct useful internal models and derive reliable predictions while remaining narrow, brittle, opaque or weakly unified.

Recognising these intermediate forms matters for both philosophy and engineering. It allows us to distinguish genuine epistemic achievement from memorisation without treating every successful model as a fully fledged scientific reasoner. It also directs attention towards the training objectives, architectures and interfaces that determine what kind of understanding a system can develop.