Researchers Identify and Address Rank Collapse Problem in Feedback Alignment Training
A new study on arXiv identifies that feedback alignment, a biologically plausible alternative to backpropagation, suffers from rank collapse that limits its effectiveness in deeper neural networks. The researchers found that feedback alignment error signals are constrained to lower-dimensional subspaces compared to standard backpropagation, restricting the network's ability to explore the parameter space. By applying orthogonalization techniques and activation normalization, they achieved significant accuracy improvements, suggesting a path toward scaling biologically plausible learning algorithms.
Researchers investigating feedback alignment (FA)—a training method that avoids backpropagation's biological implausibility by using fixed random feedback weights—discovered that the approach fails to scale to deeper architectures due to rank collapse. Through analysis of networks trained on CIFAR10, they found that FA error signals have considerably lower effective rank than standard backpropagation, confining learning to a lower-dimensional subspace and limiting parameter space exploration. To address this limitation, the team evaluated two mechanisms: Muon, an optimizer that orthogonalizes weight updates, and hidden activity normalization to promote activation orthogonality. Testing on larger architectures and benchmarks including CIFAR100 with ResNet-18, these methods consistently improved performance, with accuracy gains reaching 9 percentage points. The findings identify low-dimensional gradient dynamics as a fundamental obstacle to scaling feedback alignment and propose inducing higher-dimensional update geometry as a solution.
Limitations & open questions
The study does not discuss computational overhead or training time comparisons between feedback alignment with the proposed improvements versus standard backpropagation, which would be relevant for practical applicability. Additionally, the paper does not address whether these improvements maintain the biological plausibility advantages that motivate feedback alignment research in the first place.
What different sources said
- arXiv cs.LGCenter
Overcoming Rank Collapse in Feedback Alignment
Related
Researchers Identify D-Retro-Inverso Peptide Candidates for Treating Cardiac Amyloidosis
Researchers have designed and computationally evaluated four peptide candidates made from D-amino acids that may inhibit the formation of Serum Amyloid A (SAA) fibrils associated with cardiac complications following heart attacks. The study builds on mouse model evidence suggesting SAA aggregates contribute to long-term myocardial infarction complications and may operate similarly in humans. Two candidates, DRI-R5S and DRI-H6A, were identified as particularly promising for potential drug development.
Study Identifies D-Retro-Inverso Peptides as Potential Treatments for Cardiac Amyloidosis
Researchers designed four peptide candidates using D-amino acids to inhibit Serum Amyloid A (SAA) fibril formation, which may contribute to complications following heart attacks. The study, conducted using molecular dynamics simulations in a mouse model, identified two peptides—DRI-R5S and DRI-H6A—as promising drug candidates. This work could lead to new therapeutic approaches for cardiac amyloidosis, a serious post-infarction complication.
Scientists Discover Previously Unknown Branch of Tryptophan Metabolism in Humans
Researchers identified a new enzymatic step in human tryptophan catabolism, showing that the protein ASPDH acts as a 2-aminomuconate reductase to produce a previously unknown amino acid. This discovery fills a gap in the kynurenine pathway, one of the body's major metabolic routes for processing the amino acid tryptophan. The finding expands understanding of human metabolism and may have implications for understanding NAD cofactor production and related metabolic diseases.