LLM Inference Performance on Edge Devices: Thermal Constraints Dominate Mobile Platforms
Researchers benchmarked a quantized large language model across four consumer and edge platforms—Raspberry Pi with NPU, Samsung Galaxy S24 Ultra, iPhone 16 Pro, and a laptop GPU—finding that thermal management, not peak compute, is the primary constraint on mobile devices. The iPhone 16 Pro lost nearly half its throughput within two inference iterations, while the S24 Ultra's OS-enforced GPU frequency floor halted inference entirely. The findings highlight that real-world on-device AI performance diverges sharply from hardware specifications, with important implications for always-on personal AI agents.
A preprint study published on arXiv benchmarked Qwen 2.5 1.5B (4-bit quantized) across four platforms under sustained inference conditions: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, researchers measured throughput, latency, power draw, and thermal behavior. On mobile platforms, thermal throttling emerged as the dominant constraint: the iPhone 16 Pro shed nearly half its throughput by the second iteration, and the S24 Ultra experienced an OS-enforced GPU frequency floor that terminated inference entirely. The RTX 4050 sustained the highest throughput at 131.7 tokens per second at 34.1 W, constrained primarily by its battery power ceiling, while the Hailo-10H NPU sustained only 6.9 tokens per second but did so at under 2 W with near-zero variance. Notably, the Hailo-10H matched the RTX 4050 in energy proportionality despite 19x lower throughput, suggesting it may be well-suited for low-power always-on deployments. The authors caution that results reflect combined hardware-software platform behavior for a single model and prompt type, and should not be generalized as claims about hardware capability in isolation.
What's missing
The study tests only one model (Qwen 2.5 1.5B at 4-bit quantization) and one prompt type (258 tokens), leaving open how results would generalize to larger models, different quantization schemes, or varied prompt lengths. The study does not evaluate CPU-only inference paths on any platform, nor does it compare against other NPU-equipped edge devices.
What different sources said
- arXiv cs.AICenter
Resource-Aware LLM Reasoning for Mobile Edge General Intelligence
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.