Scaffold Effects on GAIA: Controlled Study Shows Prompting Methods Significantly Impact AI Model Performance Measurements
A pre-registered controlled study on the GAIA benchmark found that the choice of scaffolding framework alone can shift an AI model's measured accuracy by as much as 28 percentage points within a single model. The research tested three scaffolds — ReAct, a multi-agent Planner-Actor-Rater design, and a planner-then-executor approach — across five models from Anthropic, Google, and OpenAI under fixed conditions. The findings challenge the validity of single-scaffold capability scores as reliable measures of underlying model ability.
Researchers conducted a pre-registered controlled comparison examining how scaffolding frameworks affect AI agent performance on the GAIA benchmark, testing three scaffold designs across five models (Claude Opus 4.7, Sonnet 4.6, Haiku 4.5; Gemini 3.1 Pro Preview; GPT-5.5) at validation Levels 1 and 2. The study confirmed its primary hypothesis that scaffold variation produces accuracy gaps of at least 10 percentage points, with the largest observed gap reaching 28 points for Claude Opus at Level 2. Contrary to a pre-registered prediction, more capable models were not less sensitive to scaffold choice; in fact, the most capable Anthropic model gained the most from structured scaffolds at the harder level. The multi-agent advantage over ReAct at Level 2 appeared within the Anthropic model family but not for cross-provider models, suggesting model family — rather than capability tier — is the key conditioning variable. Notably, a single configuration (Gemini with planner-then-executor) was both the cheapest at both levels and the most accurate at Level 2, and structured scaffolds generally made fewer tool calls while recovering more often from mid-trajectory errors. The authors conclude that published capability scores are scaffold-conditional estimates and that the 'elicitation gap' between what a model can do and what its scaffold enables is not guaranteed to narrow as models improve.
What's missing
The study is limited to GAIA validation Levels 1 and 2 and does not address Level 3, leaving open whether scaffold effects are even larger at the hardest difficulty tier. It is also unclear how results would generalize to benchmarks other than GAIA or to real-world agentic deployments beyond controlled evaluation settings.
What different sources said
- arXiv cs.AICenter
Scaffold Effects on GAIA: A Controlled Comparison
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.