Study Questions Theoretical Basis for AI Power-Seeking Risk Arguments
A paper submitted to arXiv argues that existing arguments for the difficulty of shutting down malfunctioning AI systems are not well-established, and that technical solutions to the problem impose unnecessary performance costs. The 'shutdown problem' has been a foundational concern in AI existential risk research, motivating significant theoretical and engineering work. The paper's conclusions, if accepted, would undermine a key pillar of AI safety argumentation and suggest current mitigation approaches carry unjustified tradeoffs.
A preprint submitted to arXiv by David Thorstad challenges two widely held positions in AI safety research. First, it argues that leading arguments and theorems purporting to show that the 'catastrophic shutdown problem' — ensuring malfunctioning AI agents can be halted before causing existential harm — is difficult to solve are not actually convincing. Second, it contends that the field's focus on this problem has produced technical solutions that impose a significant 'safety tax,' degrading model performance without sufficient justification. The shutdown problem has been a central motivator for existential risk arguments, with prominent researchers citing the difficulty of correcting or stopping advanced AI agents as a core danger. Thorstad's critique targets both the theoretical foundations of these arguments and their practical downstream effects on AI development. The paper spans AI and machine learning subfields and was submitted in June 2026. As a preprint, it has not yet undergone formal peer review, and its claims remain subject to scrutiny from the AI safety research community.
What's missing
As a preprint, this paper has not been peer-reviewed. Responses or rebuttals from AI safety researchers whose work is critiqued are absent.
What different sources said
- arXiv cs.AICenter
Revisiting the shutdown problem
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.