Sci-Rho: New Multilingual Benchmark Tests AI Model Robustness on STEM Problems
Researchers introduced Sci-Rho, a dynamic benchmark with 42,420 visually-grounded STEM problems across five subjects and seven languages to test AI model robustness. The benchmark uses executable Python code to generate problem variations while keeping content equivalent, revealing significant gaps between average and worst-case accuracy across 17 state-of-the-art vision-language models. The findings highlight that smaller models degrade across languages while larger proprietary models remain more robust, suggesting current evaluation methods may not adequately measure AI reliability.
Researchers have created Sci-Rho (Science Robustness), a comprehensive benchmark designed to assess how well AI vision-language models (VLMs) handle variations of STEM problems. The benchmark comprises 4,242 problem templates across five subjects in seven languages, created by domain experts including Olympiad medalists. Each template is implemented as executable Python code that generates diverse problem instances by varying numerical values, visual patterns, geometric shapes, colors, and function types, resulting in 42,420 total instances with reasoning steps and ground-truth solutions. Evaluation of 17 state-of-the-art VLMs revealed a notable gap between worst-case accuracy (performance across all variations of a template) and average accuracy, with smaller models showing significant performance degradation across languages while larger proprietary models remained more robust. Analysis of attention mechanisms in VLMs showed substantial cross-lingual variation in how models allocate attention between image and text tokens, suggesting that current static benchmarks may not adequately capture model reliability.
What's missing
The study does not specify which five STEM subjects are covered, which seven languages are included, or provide detailed performance metrics (specific accuracy percentages) for individual models or languages. Additionally, the paper does not discuss potential limitations of the Python code generation approach or acknowledge whether the benchmark may have inherent biases in problem design across different languages and cultures.
What different sources said
- arXiv cs.CLCenter
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.