Researchers Propose Geometric Approach to Improve LLM Consistency in Open-Ended Tasks

A new preprint from arXiv proposes Embedding-Based Agreement (EBA), a method designed to extend self-consistency techniques for large language models (LLMs) beyond tasks with discrete, categorical outputs. Traditional self-consistency methods work by sampling multiple model outputs and selecting the most frequently occurring answer, which is effective for math or multiple-choice problems but breaks down for open-ended tasks. EBA addresses this by clustering sampled generations in embedding space and treating geometric proximity as a proxy for semantic agreement. Experiments across mathematical reasoning, code generation, and text summarization show that EBA consistently outperforms random selection and scales more stably than approaches based on LLM-as-judge evaluation or uncertainty estimation. The authors also find that generations concentrated near the central regions of representation space tend to be of higher quality, while peripheral generations are less accurate. Notably, the method is training-free and generalizes across model families and embedding spaces, including native hidden representations. The findings suggest that self-consistency is better understood as a geometric rather than purely symbolic property of model outputs.