Study Shows Two-Layer Linear Auto-Regressive Models Learn to Approximate Kalman Filtering
Researchers have proven theoretically that two-layer linear auto-regressive models, when trained on partially observed linear dynamical systems, naturally learn to approximate Kalman filtering without any explicit knowledge of the underlying dynamics. The Kalman filter is a classical algorithm for optimally estimating hidden states from noisy observations, and the finding shows that simple auto-regressive architectures rediscover this solution from data alone. The result offers a formal explanation for why auto-regressive models develop meaningful latent representations, with implications for understanding larger sequence models used in language and video.
A paper accepted to ICML 2026 demonstrates that two-layer linear auto-regressive models trained via empirical risk minimization on data from partially observed linear dynamical systems implicitly learn to approximate the Kalman filter. Specifically, the learned hidden representations coincide, up to a similarity transformation, with the state estimates produced by the optimal Kalman filter, even though the model has no explicit access to the system's dynamics or state. The theoretical result rests on three key insights: the Kalman filter can be well approximated by a bounded auto-regressive model, the two-layer optimization landscape is 'benign' in that all stationary points are either strict saddles or global minima despite non-convexity, and finite-sample guarantees are established for prediction error, parameter estimation error, and latent state recovery. Numerical simulations corroborate the theoretical findings. The work addresses a longstanding open question about why auto-regressive models develop structured latent representations, providing a rigorous theoretical foundation grounded in classical control theory.
What's missing
The analysis is restricted to two-layer linear auto-regressive models and linear dynamical systems; it is an open question whether analogous results extend to nonlinear systems, deeper architectures, or the large-scale transformer-based models used in practice.
What different sources said
- arXiv cs.AICenter
Two-Layer Linear Auto-Regressive Models Estimate Latent States
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.