Classical learning theory leads one to expect that a model which has fitted its training data exactly and fails on new data has overfitted, and that further training will not help. Deep networks have made that expectation unreliable more than once. They can fit randomly labelled data (Zhang et al., 2017), and their test error can fall, rise and fall again as model size or training time increases, the pattern known as double descent (Belkin et al., 2019; Nakkiran et al., 2020). Grokking is a further and especially stark departure from the textbook picture.

The term was introduced by Power et al. (2022), who trained small Transformers on tables of binary operations such as addition and division modulo a prime, holding out part of each table for validation. This note comments on that report and on the explanations that followed. It presents no new experimental results; its purpose is to gather what the published work agrees on and to state some practical points that a reader setting up such experiments may find useful.

In the original description the network reaches perfect training accuracy early, while validation accuracy stays near chance. Training continues with no visible change in either curve, and then, after a number of steps that can exceed the time to fit the training set by orders of magnitude, validation accuracy climbs to nearly perfect. Two regularities were reported alongside. The smaller the fraction of the table used for training, the longer the delay, until below some fraction generalisation does not occur within the budget. And among the interventions tried, weight decay was the most effective at shortening the delay (Power et al., 2022).

The apparent suddenness has been qualified by later work. Nanda et al. (2023) reverse-engineered a one-layer Transformer trained on modular addition and found that it computes the answer with discrete Fourier components and trigonometric identities, in effect turning addition into rotation around a circle. With that mechanism identified they defined progress measures that track it directly, and these change smoothly through the plateau. They describe three phases after the initial fit, which Figure 1 places in order: memorisation, the gradual formation of the generalising circuit, and a cleanup phase in which weight decay removes the memorising components.

Several accounts of the delay have been published, and they are more complementary than they first appear. Table 1 lists them with the mechanism each emphasises and the prediction that distinguishes it.

Liu et al. (2022) treat the problem as one of representation learning and map out regions of hyperparameter space in which a network generalises promptly, generalises late, only memorises, or fails to train. In a later paper the same group attribute the delay to a mismatch between the weight norms at which training loss and test loss are low, and show that the effect can be induced on tasks beyond arithmetic by starting from large weights (Liu et al., 2023). Varma et al. (2023) argue that the network holds two circuits, one that memorises and one that generalises, and that weight decay favours whichever produces the required outputs at lower parameter norm. Since memorising more examples costs more norm and generalising does not, there is a dataset size above which the generalising circuit wins.

Thilak et al. (2022) report the effect without explicit weight decay and connect it to a recurring instability of adaptive optimisers. A related line of theory shows that gradient descent on separable data keeps moving towards a large-margin solution long after training error is zero, and does so slowly (Soudry et al., 2018). What these accounts share is the view that a simpler solution is preferred, by explicit or implicit regularisation, and takes much longer to reach than the memorising one.

Four practical points follow from reading these papers together. The first is that the delay is a property of a configuration and not of the task alone. Training fraction, the strength of weight decay, the scale of initialisation and the choice of optimiser each move the onset of generalisation, and a small change can turn a long delay into none or into a run that never generalises. Weight decay should be specified exactly, including whether it is decoupled from the adaptive update as in AdamW (Loshchilov and Hutter, 2019), because the two forms regularise differently.

The second point is that loss and accuracy are the least informative quantities to record. Total weight norm, the norm of individual layers, and where a mechanism is known, measures tied to it, reveal movement during the plateau that the headline curves hide. The third is a matter of presentation: because the delay spans orders of magnitude, curves should be drawn on a logarithmic step axis, and a linear axis exaggerates the abruptness of the transition. The fourth is that several seeds are needed before any statement about timing is made, since the step at which generalisation appears varies between runs with identical settings.

Modular arithmetic is attractive because the full input space is small, the correct algorithm is known, and a one-layer network can be understood completely. The same features limit what can be inferred. The task has an exact, compact solution, and natural data rarely do; the data are free of label noise; and the networks are far smaller than those in use. Barak et al. (2022) found a comparable pattern of hidden progress when networks learn sparse parities, which suggests the behaviour is not peculiar to one task, but that is still an algorithmic problem.

It would therefore be unsafe to conclude that large models trained on language or images will generalise if only they are trained long enough. The better-supported lesson is narrower. A flat validation curve does not prove that nothing is changing inside the network, and early stopping on such a curve can end training before a slower, more general solution has formed.

Grokking on modular arithmetic is now among the best-understood training phenomena in deep learning, largely because the task is small enough to be taken apart. The published explanations converge on a competition between a quick memorising solution and a slower, more economical generalising one, with regularisation deciding the outcome. For those who study it, the configuration matters as much as the task, internal measures matter more than accuracy, and caution is warranted in carrying conclusions to larger systems.

To be declared by the author(s) before publication.

To be declared by the author(s) before publication.