Stroop Task Reveals Significant Attention Control Limitations in Leading AI Models

A study led by Suketu Patel administered the Stroop task—a well-established psychological test where participants identify ink colors of words that may conflict with the word's meaning—to several leading large language models. The AI systems performed well on short lists of five words but experienced dramatic accuracy declines as list length increased: GPT-4o dropped from 91% accuracy to 57% at ten words and 15% at forty words, while Claude 3.5 Sonnet fell to 24% accuracy with forty-word lists. When matching and mismatched color words appeared together, performance deteriorated further, with some models showing near-zero accuracy on mismatched items. The researchers found that AI models increasingly defaulted to reading the words themselves rather than identifying ink colors, suggesting they could not consistently suppress their most heavily trained responses. This contrasts sharply with humans, who despite being naturally biased toward reading words can maintain high accuracy across long lists of conflicting stimuli, indicating a fundamental difference in how biological brains and current AI systems manage attention and executive control.