A Suite of LMs Comprehend Puzzle Statements as Well or Better Than Humans

TY - JOUR

T1 - A Suite of LMs Comprehend Puzzle Statements as Well or Better Than Humans

AU - Rakshit, Supantho

AU - Hu, Jennifer

AU - Mahowald, Kyle

AU - Goldberg, Adele E.

N1 - Publisher Copyright: © 2026 Supantho Rakshit, Jennifer Hu, Kyle Mahowald, and Adele E. Goldberg. Published under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.. This is an open-access article distributed under the terms of the https://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.

PY - 2026

Y1 - 2026

N2 - This paper reexamines a recent claim that Large Language Models lag behind humans in language comprehension on what were described as minimally complex statements. We argue that human performance was overestimated and LM performance, underestimated. Moreover, both people and lower-performing LMs are disproportionately challenged by queries involving potentially appropriate inferences, suggesting shared pragmatic sensitivity rather than model-specific deficits. Analysis of more sensitive log probabilities of Llama-2-70B demonstrate ceiling-level accuracy and pragmatic sensitivity. A separate group of LM grammaticality judgments previously characterized as incorrect are shown to correlate with human judgments, while certain reasoning models approximate idealized judgments when prompted to respond as an expert generative syntactician. Overall, the findings suggest that apparent deficits in LM performance may reflect task design, evaluation choices, and assumptions about human performance, rather than deficiencies in current models.

AB - This paper reexamines a recent claim that Large Language Models lag behind humans in language comprehension on what were described as minimally complex statements. We argue that human performance was overestimated and LM performance, underestimated. Moreover, both people and lower-performing LMs are disproportionately challenged by queries involving potentially appropriate inferences, suggesting shared pragmatic sensitivity rather than model-specific deficits. Analysis of more sensitive log probabilities of Llama-2-70B demonstrate ceiling-level accuracy and pragmatic sensitivity. A separate group of LM grammaticality judgments previously characterized as incorrect are shown to correlate with human judgments, while certain reasoning models approximate idealized judgments when prompted to respond as an expert generative syntactician. Overall, the findings suggest that apparent deficits in LM performance may reflect task design, evaluation choices, and assumptions about human performance, rather than deficiencies in current models.

KW - LLM and human alignment

KW - LLM evaluation

UR - https://www.scopus.com/pages/publications/105037749119

UR - https://www.scopus.com/pages/publications/105037749119#tab=citedBy

U2 - 10.1162/OPMI.a.344

DO - 10.1162/OPMI.a.344

M3 - Article

C2 - 41953133

SN - 2470-2986

VL - 10

SP - 431

EP - 440

JO - Open Mind

JF - Open Mind

ER -