FOGO: New Optimizer Addresses Gradient Interference in Standard and Continual Learning
Researchers have introduced FOGO, a gradient optimization algorithm designed to prevent both short-term and long-term knowledge forgetting during neural network training. The method works by orthogonalizing momentum updates and storing representative past gradient directions in a compact memory structure to resolve conflicts at each training step. If validated broadly, FOGO could improve training efficiency and knowledge retention across a wide range of machine learning tasks, from continual learning to large language model pretraining.
A preprint posted to arXiv proposes FOGO (Forgetting-aware Orthogonalization Optimizer), a new optimization algorithm that reframes 'forgetting' as a general phenomenon in gradient-based training rather than a problem unique to continual learning. The authors argue that dominant mini-batch gradients routinely suppress rarer but useful update directions at every training step, and when those directions are never revisited, losses accumulate into the long-term forgetting seen in continual learning settings. FOGO addresses this by spectrally orthogonalizing momentum updates so that no single gradient direction monopolizes optimization, and by maintaining a compact codebook memory of past gradient directions using random projections that provably preserve pairwise distances in low-dimensional space. At each step, conflicts between the current update and stored directions are resolved through lightweight orthogonal correction followed by a proximal step, adding minimal computational overhead and requiring no raw data storage. The authors report that FOGO outperforms Adam and Muon optimizers across several benchmarks, including class-imbalanced classification, continual visual learning under domain and class shifts, continual fine-tuning of LLaVA-7B, and GPT-2 pretraining.
What's missing
As a preprint, this work has not yet undergone peer review. Computational cost comparisons beyond the claim of 'minimal overhead' are not detailed in the abstract, and the codebook memory's behavior under very long training horizons or extremely large models is an open question.
What different sources said
- arXiv cs.LGCenter
FOGO: Forgetting-aware Orthogonalization Optimizer
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.