Marianne de Heer Kloots

@mdhk.net

1338 Followers 554 Following

Linguist in AI & CogSci 🧠👩‍💻🤖 PhD student @illc-uva.bsky.social 🌐 https://mdhk.net/ 🐘 https://scholar.social/@mdhk 🐦 https://twitter.com/mariannedhk

Cover page of our commentary. Title: Linguists should learn to love speech-based deep learning models Authors: Marianne de Heer Kloots, Paul Boersma, Willem Zuidema Abstract: Futrell and Mahowald present a useful framework bridging technology-oriented deep learning systems and explanation-oriented linguistic theories. Unfortunately, the target article's focus on generative text-based Large Language Models (LLMs) fundamentally limits fruitful interactions with linguistics, as many interesting questions on human language fall outside what is captured by written text. We argue that audio-based deep learning models can and should play a crucial role.
Sigmoid curves fitted to the best-layer scores for all six models (two seeds for each architecture). Different levels of linguistic structure consistently show distinct learning dynamics across architectures and model seeds, with increased parallelism between levels for HuBERT's second iteration as compared to HuBERT's first iteration and Wav2Vec2 models.
Figure displaying learning trajectory results (i.e. the evolution of probe scores across training checkpoints) and a single model (Wav2Vec2, seed 2). The left column shows the best-layer scores across training steps, and the right column shows heatmaps of all layerwise scores by training steps.
Figure displaying layerwise probe results for 3 models of different architectures (Wav2Vec2, HuBERT's first iteration, HuBERT's second iteration).
Animation showing the development of syllable type structure in the sixth Transformer layer of a Wav2Vec2 model, across training checkpoints. The figure shows three-dimensional projections of model representations, which are optimized by Linear Discriminant Analysis for separability between syllable types. The projections become increasingly structured over training, with distinct clusters emerging for different syllable types.
Figure illustrating the study's methods: We train a set of six models on the same dataset, consisting of 831 hours of Dutch speech recordings (sourced from the Spoken Dutch Corpus, Multilingual LibriSpeech, and CommonVoice). We explore how results vary between model architectures with minimal differences in training set-up (Wav2Vec2, HuBERT's first iteration, HuBERT's second iteration). We probe each model's internal representations for nine types of linguistic structure, which differ in their degrees of abstraction from the acoustic signal and in their timescales of information integration. We compare results for each structure across model layers as well as training steps.
An animation illustrating 9 levels of structure in speech signals (acoustic, phonetic, syllable form, syllable type, word form, homophone disambiguation, semantic alignment, part-of-speech, and syntactic dependencies). On the right side, sigmoid curves track the development of each type of structure across training steps of 6 self-supervised speech models (Wav2Vec2, HuBERT's first iteration, and HuBERT's second iterations; two seeds each).
Poster title: Does multimodal pre-activation influence linguistic expectations in LLMs and humans? Authors: Sasha Kenjeeva, Giovanni Cassani, Noortje Venhuizen, Afra Alishahi
Commentary title: Linguists should learn to love speech-based deep learning models Authors: Marianne de Heer Kloots, Paul Boersma, Willem Zuidema Abstract: Futrell and Mahowald present a useful framework bridging technology-oriented deep learning systems and explanation-oriented linguistic theories. Unfortunately, the target article's focus on generative text-based LLMs fundamentally limits fruitful interactions with linguistics, as many interesting questions on human language fall outside what is captured by written text. We argue that audio-based deep learning models can and should play a crucial role.
Word Error Rate results for models fine-tuned for Dutch ASR (speech-to-text transcription), across 4 models and 5 evaluation datasets.
Layerwise phonetic and lexical analyses, across a read speech (MLS, top row) and a dialogue (IFADV, bottom row) dataset of spoken Dutch. Measures marked * involve optimized linear transforms, whereas others are computed zero-shot; shading indicates 95% confidence intervals. The Dutch Wav2Vec2-NL model achieves highest scores across most analyses of Dutch phone and word encoding, though the size of this language-specific advantage varies considerably across analyses.
The model comparison set includes Wav2Vec2-NL and 3 other existing Wav2Vec2-base models: facebook's multilingual voxpopuli model, facebook's English base model, and another model trained on nonspeech acoustics. The set of analysis techniques includes probing classifiers (logistic regression), ABX similarities, PCA clustering, LDA clustering, and representational similarity analysis (RSA). Word- and phone-level embeddings were created by mean-pooling model frame embeddings within words and phones respectively. The SSL-NL evaluation dataset is a curated dataset of Dutch speech recordings and accompanying forced alignments, across two domains: audiobooks (MLS) and face-to-face conversations (IFADV).
Interspeech paper title: What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training Authors: Marianne de Heer Kloots, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, Martijn Bentum
The BLiMP-NL dataset consists of 84 Dutch minimal pair paradigms covering 22 syntactic phenomena, and comes with graded human acceptability ratings & self-paced reading times. An example minimal pair: A. Ik bekijk de foto van mezelf in de kamer (I watch the photograph of myself in the room; grammatical) B. Wij bekijken de foto van mezelf in de kamer (We watch the photograph of myself in the room; ungrammatical) Differences in human acceptability ratings between sentences correlate with differences in model syntactic log-odds ratio scores.
red square against budget cuts in higher education
poster announcing the protest on November 25th, 1pm, Malieveld, The Hague
paper title: Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0
Figure with two scatterplots, showing a strong correlation between dependency accuracy (Labelled Attachment Score) and brain alignment (Representational Similarity Score) on the left, and no correlation in a scrambled control condition on the right.
Dependency parse for the sentence "De overtreder die de smeris ontvlucht was is een kronkelig paadje ingerend" (Dutch for: "The offender who had escaped from the cop ran into a winding path"). The picture shows all dependency links between the words of the sentence, such as the links between verbs and their subjects (Subj) and nouns and their determiners (Det).
Paper title ('Language Models That Accurately Represent Syntactic Structure Exhibit Higher Representational Similarity To Brain Activity') and overview figure