Study Finds MLP Residual Networks Implement Renormalization Group-Like Coarse-Graining
Researchers studying MLP residual networks trained on synthetic Markov chain sequences found quantitative evidence that these networks implement a selective coarse-graining procedure analogous to renormalization group flows in physics. The study measured effective rank collapse in the residual stream, showing that networks preserve task-relevant degrees of freedom while discarding irrelevant information based on input spectral properties. This provides the first empirical verification of a long-theorized analogy between deep neural network behavior and renormalization group dynamics.
A new arXiv preprint presents the first quantitative evidence that MLP residual networks implement a renormalization group (RG)-like structure during forward passes. The researchers trained pure MLP residual stacks on masked token prediction using synthetic Markov chain sequences with known spectral properties, allowing controlled experimental variation. They measured three key findings: (1) effective rank of the residual stream decreases monotonically with depth, consistent with progressive integration of irrelevant information; (2) this rank collapse is selective and depends on input correlation length—occurring for short-correlation chains but absent for long-correlation chains—demonstrating that networks preserve exactly the degrees of freedom relevant to the prediction task; and (3) inter-layer kernel drift concentrates at specific transitions with the remainder of the network near fixed points. Together, these findings provide position-level evidence that residual networks perform selective coarse-graining governed by input spectral structure, moving beyond previous qualitative analogies to quantitative, empirically verified predictions.
Limitations & open questions
The study's scope is limited to simple MLP architectures on synthetic Markov chain data; generalization to more complex architectures (transformers, CNNs) and real-world datasets remains unclear. The paper does not discuss computational implications or whether this RG structure emerges during training or is present from initialization.
What different sources said
- arXiv cs.LGCenter
Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks
Related
Study suggests asexual reproduction slowed early animal evolution during Ediacaran period
Researchers from the University of Cambridge found that early animals during the Ediacaran period (635-539 million years ago) reproduced asexually through runners, which limited competition and slowed evolutionary diversity. The study used fossil analysis, spatial modeling, and artificial intelligence to examine ancient ecosystems at Mistaken Point in Newfoundland. The findings help explain why animal diversity remained limited for millions of years before a dramatic burst of innovation in the Cambrian period.
UK Science Facilities Face £162m Funding Crisis With Potential Closures
Britain's world-leading science facilities, including the Diamond Light Source and ISIS Neutron and Muon Source, face potential closure or significant cuts due to a £162m funding crisis at the Science and Technology Facilities Council caused by rising electricity costs, staff expenses, and foreign exchange pressures. These facilities serve hundreds of companies and thousands of scientists domestically and internationally, with Diamond producing light 10 billion times brighter than the sun for materials research. Scientists and research leaders warn that short-term funding cuts could cause decades-long damage to the UK's scientific capability and international competitiveness.
Mitochondrial ROS Signaling Drives Avoidance Learning in C. elegans
Researchers discovered that reactive oxygen species (ROS) produced by mitochondria in postsynaptic neurons are necessary and sufficient to drive avoidance learning in C. elegans, using optogenetic stimulation of nociceptive neurons. The study demonstrates that activity-dependent mitochondrial ROS production increases glutamate receptors at synapses and strengthens neural circuits controlling avoidance behavior. This finding reveals a novel molecular mechanism linking neuronal activity to synaptic plasticity and behavioral learning.