Neural Networks Learn Circular Geometry for Modular Arithmetic, Not Traditional Neural Collapse

Researchers have submitted a preprint to arXiv challenging the universality of neural collapse (NC), a widely studied phenomenon in which terminal-layer representations of a balanced classifier converge to a simplex equiangular tight frame (ETF). In modular addition tasks, the study finds that networks instead compress representations into a rank-2 cyclic geometry, with both classifier weights and token embeddings lying on circles. The authors formalize a layerwise training mechanism in which dense cross-entropy gradients drive classifier weights into a rank-2 equiangular configuration before upstream embeddings fully reorganize, after which backpropagated gradients lock embeddings into the same plane. They further show that the resulting in-plane dynamics can be interpreted through an entropy-regularized transport framework on the unit circle, reducing embedding formation to phase alignment whose minimizers are single-frequency characters of the cyclic group. Crucially, the paper quantifies the competitive advantage of the cyclic solution: while a simplex ETF gains only a constant advantage in cross-entropy, the cyclic rank-2 solution enjoys a Θ(K) advantage under weight-decay regularization, implying a critical regularization threshold of Θ(1/K) below which the cyclic geometry dominates. These results provide a unified explanation for both the ordering of weight and embedding reorganization and the grokking phenomenon in modular arithmetic, framing generalization as a task-structured trade-off among separation, symmetry, and representational complexity.