LEDGER: New Benchmark Dataset for Evaluating LLMs on Financial Document Analysis
Researchers have published LEDGER, a benchmark dataset of 4,999 digitized corporate annual reports designed to evaluate large language models on long-context financial document understanding. Unlike existing resources that rely on plain-text SEC 10-K filings with limited question-answer pairs, LEDGER includes full documents with figures, tables, and narrative text, paired with 31 financial KPIs and market reaction data. The benchmark addresses a growing need for rigorous evaluation of AI models on complex, real-world financial documents as long-context capabilities of LLMs rapidly advance.
LEDGER (Long-context Evaluation of Documents for Grounded Extraction and Retrieval) is a newly released corpus comprising 4,999 digitized corporate annual reports, each labeled with 31 consolidated financial key performance indicators (KPIs) linked to market reactions at earnings dates. The dataset supports three distinct evaluation benchmarks of varying difficulty: a page-level KPI retrieval task with TREC-style relevance judgments covering 118,048 natural language questions, a conversational needle-in-a-haystack single-value lookup, and a full KPI extraction task from long, numerically dense documents. The authors argue that existing public financial benchmarks are inadequate because they reduce the problem to plain-text regulatory filings with only a handful of question-answer items, failing to capture the complexity of real annual reports. LEDGER also includes human OCR-quality annotations with inter-annotator agreement scores and a complete extraction, validation, and scoring toolchain to support reproducible research. As a demonstration of the dataset's utility, the researchers conducted a case study examining the relationship between CEO-letter rhetoric and post-publication market impact.
What's missing
The study's own scope is limited to a 5-page paper with a single figure, suggesting the full methodology, inter-annotator agreement statistics, and baseline model performance results may not yet be comprehensively reported. The case study linking CEO rhetoric to market impact is correlational and its causal interpretation remains an open question.
What different sources said
- arXiv cs.CLCenter
LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction
Related
Gut Bacteria Enzyme Found to Break Down Heat-Processed Food Compounds, Producing Novel Biogenic Amines
Researchers have discovered that an enzyme in common gut bacteria can degrade N-epsilon-carboxymethyllysine (CML), a compound formed during thermal food processing, producing previously unknown biogenic amines. The enzyme, ornithine decarboxylase SpeC from enterobacteria, acts on CML and related modified lysine derivatives through a low-level 'underground' catalytic activity. This finding suggests a previously unrecognized communication axis between thermally processed dietary compounds and gut microbial physiology, with potential implications for host health.
Full-Length Gene Sequencing Reveals Two Distinct Bacterial Communities in Black-Legged Ticks Expanding Into Canada
Researchers used Oxford Nanopore full-length 16S rRNA gene sequencing to characterize the microbiome of Ixodes scapularis black-legged ticks collected in Nova Scotia, Canada, distinguishing between tick-adapted bacteria and environmentally acquired bacteria. The study comes as I. scapularis — the primary vector of Lyme disease — is rapidly expanding northward into Canada due to climate change. The findings suggest that environmentally derived bacteria in tick microbiomes are not mere contamination, which has implications for how tick microbiome data is collected and interpreted across surveillance studies.
Study Identifies Metabolic Link Between Cell Envelope Stress and Biofilm Formation in Bacteria
Researchers have discovered that the metabolite acetyl-CoA directly inhibits enzymes that degrade the bacterial signaling molecule c-di-GMP, connecting cell envelope biosynthesis stress to biofilm formation in Pseudomonas aeruginosa. The study found that sub-inhibitory concentrations of antibiotics targeting early peptidoglycan biosynthesis — but not other antibiotic classes — elevate c-di-GMP levels by reducing phosphodiesterase activity, with acetyl-CoA competing for the enzyme active site. Because the relevant enzyme domain is broadly conserved across bacterial species, this checkpoint mechanism may be widespread and could have implications for understanding antibiotic-induced biofilm responses.