Study Finds Instruction Files for AI Agents Show Mixed Results in Pull Request Success
A new study analyzing over 15,500 agentic pull requests finds that providing instruction files to AI coding agents does not reliably improve their performance. Researchers examined 148 software projects before and after instruction file creation, measuring merge rates, code complexity, and review effort. The findings suggest that how instruction files are written matters significantly, with longer, well-structured files correlating with better outcomes.
Researchers from the 23rd International Conference on Mining Software Repositories analyzed 15,549 agentic pull requests across 148 GitHub projects to assess whether instruction files — documents that guide AI agents like GitHub Copilot on project navigation, testing, and best practices — actually improve AI-generated code contributions. The study found highly variable results: 27.7% of projects saw their merge rate increase by at least 20% after introducing instruction files, while 26.35% experienced a decrease. Similar inconsistency was observed across other metrics, including code churn, number of modified files, time to merge, and review comment volume. A preliminary exploration revealed that projects achieving higher merge rates tended to use substantially longer instruction files organized into more sections and subsections, suggesting that file quality and structure are key factors. The authors propose treating instruction file development as a formal software engineering discipline — termed 'Instructions-as-Code' — and call for further research to help practitioners write more effective guidance for AI agents.
What's missing
The study is observational and cannot establish causation between instruction file characteristics and improved outcomes; confounding factors such as project size, team experience, or task difficulty are not fully controlled for. The analysis relies on a single dataset (AIDev) and focuses on GitHub Copilot-style agents, so generalizability to other AI coding tools is unclear. The paper notes the instruction file quality findings are from a 'first exploration,' meaning they are preliminary and not yet rigorously validated.
What different sources said
- arXiv cs.AICenter
Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.