Scaling laws in deep learning typically describe how loss decreases as compute increases, following a curve whose exponent is usually treated as fixed for a given architecture family. This paper challenges that assumption, showing that certain architectural interventions can directly shift this exponent, producing performance gains that compound with additional compute rather than simply offsetting it by a constant factor.
The authors build their argument around looped transformers, an architecture where the same computational block is repeated multiple times, a mechanism usually associated with reasoning rather than training dynamics. By increasing the number of loops during training as a way to grow the model, with or without shared weights, they observe the largest shifts in scaling exponents. A 7.4B parameter model built this way matches the performance of a 13B GPT-3 model on the CORE benchmark while using roughly twenty times less compute.
Compute efficiency gains grow with scale, setting this approach apart from typical optimizations that deliver constant or diminishing returns. The researchers also test a simpler boundary operator, which normalizes and re-injects an earlier block into a standard, non-looped transformer; this technique yields similar but more modest gains. In data-constrained settings where models see the same examples multiple times, standard looping also acts as a useful regularizer, and it becomes compute-optimal to increase the number of loops as model scale grows.
The authors frame these results through the lens of usable computational depth: for a fixed compute budget, the goal is to make better use of a transformer's effective depth. This framing suggests a research direction distinct from simply scaling parameters or data, pointing instead to the internal structure of computation itself as a lever for efficiency, one that could inform how future large-scale models are designed.