DecompSR: New Benchmark Dataset for Testing Compositional Spatial Reasoning in Large Language Models
Researchers have released DecompSR, a large benchmark dataset of over 5 million datapoints designed to evaluate compositional multihop spatial reasoning in large language models. The dataset is procedurally generated and verified correct by construction using a symbolic solver, allowing independent manipulation of four compositionality dimensions: productivity, substitutivity, overgeneralisation, and systematicity. Benchmarking results show LLMs perform poorly on productive and systematic generalisation in spatial reasoning while remaining relatively robust to linguistic variation, highlighting a specific gap in current AI capabilities.
DecompSR (Decomposed Spatial Reasoning) is a newly introduced benchmark dataset and generation framework containing over 5 million datapoints aimed at rigorously probing compositional spatial reasoning in large language models. The dataset is built procedurally and is guaranteed correct by construction, with its correctness independently verified through a symbolic solver — a methodological strength that distinguishes it from many existing benchmarks. Researchers designed DecompSR to allow independent variation of four key aspects of compositionality: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order and distractors), and systematicity (novel linguistic elements). Comprehensive benchmarking across multiple LLMs reveals that models struggle most with productive generalisation — tasks requiring deeper chains of spatial reasoning — and with systematic generalisation to novel linguistic elements. In contrast, LLMs show greater robustness when faced with linguistic variation in how spatial problems are expressed. The framework's ability to isolate individual compositionality dimensions provides a finer-grained diagnostic tool than prior spatial reasoning benchmarks. The work contributes both a reusable dataset and a generation pipeline that can be extended for future evaluations.
What's missing
The paper does not report absolute performance scores or model-specific results in the abstract, making it difficult to assess the magnitude of LLM failures. It is also unclear whether the benchmark has been evaluated against non-LLM baselines (e.g., symbolic or neuro-symbolic systems) that might contextualise the results. The scope of 'a host of LLMs' benchmarked is unspecified in the abstract, leaving open questions about which model families or sizes were tested and whether findings generalise across model scales.
What different sources said
- arXiv cs.AICenter
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.