Researchers Propose Dynamical Isometry Method to Preserve Neural Network Plasticity in Continual Learning

Researchers have linked the well-known problem of plasticity loss in continual deep learning to the empirical Neural Tangent Kernel, proposing dynamical isometry — the condition that singular values of layer-wise Jacobians remain close to one — as the key mechanism for maintaining a network's ability to keep learning over time. The paper revisits a class of networks that are almost-everywhere isometric while still serving as universal Lipschitz function approximators, demonstrating that near-isometry does not sacrifice expressive nonlinear representations. For general architectures, the authors propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units, a known contributor to plasticity loss. Building on these insights, they introduce AdamO, an adaptive optimizer analogous to AdamW that separates isometry regularization from gradient-based updates. The framework also reinterprets existing plasticity-preserving methods, showing they capture only partial measures of isometry. Benchmarks spanning both supervised and reinforcement learning continual-learning settings show that the proposed methods consistently match or outperform prior approaches. The work has been accepted at the Forty-Third International Conference on Machine Learning (ICML 2026).