Sathvik

@sathvik.bsky.social

276 Followers 268 Following

computational psycholinguistics @ umd he/him

Title & Abstract: Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities Sathvik Nair & Colin Phillips Commentary on Futrell, R. & Mahowald, K. (in press). How Linguistics Learned to Stop Worrying and Love the Language Models. Behavioral and Brain Sciences. http://doi.org/10.1017/S0140525X2510112X Abstract Under the lens of Marr’s levels of analysis, we critique and extend the authors’ two points about language models (LMs) and language processing: first, predicting upcoming linguistic information based on context is key to language processing, and second, that many advances in psycholinguistics would be impossible without LLMs. We also outline directions combining LLMs’ strengths with psycholinguistic models.
Replicating the cross-validated analysis of predictive power separately for whole and split words in the Natural Stories corpus under the different word segmentation methods. The predictive power of surprisal over words split into BPE tokens is significantly lower than words that were treated as whole. This is not the case for words segmented into morphological units.
Violinplot with distributions of surprisal of words with different numbers of subword tokens, split by corpus and segmentation method. On top of the plot, is a dotted red line that highlights that surprisal over morphological units increases incrementally, but surprisal over BPE tokens sharply increases and then plateaus for words broken into subword tokens.
Distribution of predictive power of surprisal under models trained under each tokenization scheme, evaluated using 10-fold cross-validation for each corpus. There is no major difference in predictive power associated with tokenization.