Lecture by Samuel Joseph Amouyal (Collaboration with Abeer Assi and Prof. Aya Meltzer-Asscher)

A joint work by Samuel Joseph Amouyal, Abeer Assi, Prof. Aya Meltzer-Asscher, and Prof. Jonathan Berant:

While the human misunderstood the LLM did too. Comparing humans and LLMs on Garden Paths

(Delivered at ISCOP 2025)

Abstract:

Research on object/subject garden path (GP) sentences with optionally transitive (OPT) verbs, such as “while the man hunted the deer ran into the woods” has revealed consistent misinterpretations, where readers mistakenly respond “yes” to questions like “did the man hunt the deer?”.. Our study examined how the plausibility of the initial interpretation and the transitivity bias of the OPT verb affect misinterpretation, exploring the similarities and differences between humans and Large Language Models (LLMs), with the latter performing also paraphrasing and drawing. In our first experiment (N=3600), we varied sentence structure (GP vs. non-GP), plausibility (plausible vs. implausible initial parsing), and the transitivity degree of OPT verbs (ranging from intransitively-biased to transitively-biased verbs). The second experiment (N=960) focused on sentences with reflexive (e.g,. “dressed”) and alternating unaccusative verbs (e.g., “changed”), which do not require an object for full comprehension. Human results showed better comprehension for non-GP compared to GP and for implausible compared to plausible initial parses, with the largest gains observed in reflexive sentences. Transitivity bias affected reanalysis, as highly transitive verbs diminished the impact of structure by construing the NP as an object regardless of its position. Comparisons revealed that LLMs encounter difficulties with similar sentence types as humans across all tasks. Our findings in humans (confirmed the presence of partial reanalysis in GP sentences), supporting “good enough” approaches to sentence processing. The parallels in comprehension behavior between LLMs and humans suggest promising research avenues into understanding sentence processing mechanisms across both modalities.

Congratulations!