Still: New Method for Efficient KV Cache Compression in Large Language Models
Two separate research efforts published this week address major computational bottlenecks in large language model inference: one introduces Latent Context Language Models (LCLMs) that compress input context up to 16x before decoding, while another proposes Semantic Cache Distillation (SCD) to speed up KV cache transfer in disaggregated serving. LCLMs, developed by a multi-university team, achieve 8.8x inference speedup over KV cache baselines at 16x compression with accuracy remaining above 75% on long-context benchmarks; SCD delivers up to 2.65x time-to-first-token speedup while keeping generation quality within 5% F1 of a full-cache oracle. Both approaches target production deployment, with LCLMs open-sourced on HuggingFace and SCD accepted to ICML 2026.
As large language model deployments scale, two distinct infrastructure bottlenecks have emerged: the cost of processing ever-longer context windows, and the latency of transferring KV caches between disaggregated serving nodes. A multi-institution team from NYU, Columbia, Princeton, University of Maryland, Harvard, and Lawrence Livermore National Laboratory proposes LCLMs, an encoder-decoder architecture pairing a 0.6B encoder with a 4B decoder that compresses input token sequences into latent embeddings before the decoder ever runs, achieving 4x compression with only a 2.65 percentage-point accuracy drop on the RULER benchmark and 16x compression with an 8.8x throughput gain over KV cache baselines. Separately, researchers behind Semantic Cache Distillation (SCD), accepted to ICML 2026, address the problem of transmitting large KV caches across heterogeneous model variants in disaggregated serving by replacing raw cache transmission with compact semantic codes reconstructed via low-rank subspaces, with sparse corrective patches at transition layers to prevent error accumulation. SCD achieves up to 2.65x TTFT speedup in bandwidth-constrained regimes while staying within 5% F1 of oracle quality. The LCLM work was trained on over 350 billion tokens and is designed to integrate directly into existing RAG and agentic pipelines, though the researchers acknowledge that compressing live reasoning traces remains an open problem. Both efforts represent a shift from post-hoc cache pruning toward architectural solutions that reduce compute and memory demands earlier in the inference pipeline.
What's missing
For SCD, the paper does not detail how the low-rank reconstruction quality degrades as the degree of fine-tuning divergence between base and adapted models increases, which is a practical concern for real-world heterogeneous deployments.
What different sources said
- arXiv cs.CLCenter
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
- VentureBeatCenter
Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.