Dᴜsᴛɪɴ Sᴛᴏʟᴛᴢ

@dustinstoltz.com

230 Followers 272 Following

Sociologist | Computational Social Science | Culture and Cognition www.dustinstoltz.com Mapping Texts: www.textmapping.com Mastodon: fediscience.org/@dustinstoltz

Currently, there are thousands of large pretrained language models (LLMs) available to social scientists. How do we select among them? Using validity, reliability, reproducibility, and replicability as guides, we explore the significance of:(1) model openness, (2) model footprint, (3) training data, and (4) model architectures and fine-tuning. While ex ante tests of validity (i.e., benchmarks) are often privileged in these discussions, we argue that social scientists cannot altogether avoid validating computational measures (ex-post). Replicability, in particular, is a more pressing guide for selecting language models. Being able to reliably replicate a particular finding that entails the use of a language model necessitates reliably reproducing a task. To this end, we propose starting with smaller, open models, and constructing delimited benchmarks to demonstrate the validity of the entire computational pipeline.
This paper outlines a distributional approach to institutional analysis, reconceptualising institutions as distributions of knowledge and activity across people. We argue that institutionalisation and institutional change are best understood by focussing on actors with the requisite knowledge and motivation to keep institutional patterns going, fix them when they go awry, or transform them when required, here called functionaries. The distributional approach allows us to distinguish between two main types of institutional change often conflated in the literature: Content-based and formal change. Content-based change, the one most often discussed, involves the importation, recombination, or expansion of specific patterns of activity. In contrast, formal change, often neglected in the literature, refers to shifts in the distribution of knowledge and activity, leading to dynamics of centralisation and decentralisation of institutional patterns. In this way, the distributional approach highlights the role of functionaries in both institutional stability and change, providing a micro-level perspective on institutional dynamics.
The image includes a mock up for the book "Mapping Texts: Computational Text Analysis for the Social Sciences" with a text box at the top saying "Second edition in the works!" and another text "With 100% more Al*." But this is not referring to artificial intelligence, but rather to an artist that is halfway to an EGOT: Weird "Al" Yankovic. An image of Homer Simpson is on the phone, presumably hearing about this new edition, and is so excited he exclaims "Whoo-hoo!" with a triumphant fist pump.