Andrew Lampinen

@lampinen.bsky.social

8168 Followers 731 Following

Interested in cognition and artificial intelligence. Researcher at Anthropic; previously DeepMind, cognitive science at Stanford. Posts are mine. lampinen.github.io

"Reasoning comes in many technically defined forms(opens a new tab), but the basic procedure is easily recognizable: arriving at a sound conclusion by linking together intermediate steps that logically follow from each other. We do this with thoughts; LRMs use so-called chains of thought [...]" from https://www.quantamagazine.org/is-ai-reasoning-right-for-the-wrong-reasons-20260731/
Building from naturalistic experiments to cognitive theories: An overview of how we can develop theories with potentially opaque models and unnatural manipulations of natural data: building task-performing models that can reproduce behavior across naturalistic and unnatural stimuli, while tightly coupling them to reductive explanations.
How cognitive scientists can systematically build generalizable models of cognition that explain human behavior on naturalistic data. Our key proposal is leveraging dynamic meta-benchmarks (§5.1) where (a) individual benchmarks aggregate human data from experiments studying facets of a cognitive capacity. The benchmarks evolve as researchers add new facets—for example, extending object classification benchmarks to test whether classification relies on shape versus texture, then making stimuli (at least partially) reflect real-world data distributions. (b) Models are developed with generalization as the primary goal (§5.2): the same model should predict human behavior across all benchmark facets. (c) To facilitate frictionless reproducibility (§5.3), models are evaluated through a standardized interface. (d) This makes it easy for researchers to comprehensively evaluate their hypothesis (i.e. model) against a large swath of human behavioral data capturing different facets of a cognitive capacity. The best hypothesis is then the one that captures human behavior over the most human data.
Two examples of naturalistic computational cognitive science. Given a computational theory, researchers generate initial predictions to test their theory in a simplified setting. Theories are validated by their ability to predict human behavior in increasingly naturalistic conditions, and refined where they fail to predict correctly. (Top) Progression of theories of sequential task generalization. (Bottom) Examples of progression in understanding PRC involvement in difficult visual judgments..
From population variance explained, PCA (or clustering), or unit-level analyses, the model seems to strongly represent an easy feature and barely represent a harder one — even though it is linearly outputting both from this representation layer.
What do representations tell us about a system? Image of a mouse with a scope showing a vector of activity patterns, and a neural network with a vector of unit activity patterns Common analyses of neural representations: Encoding models (relating activity to task features) drawing of an arrow from a trace saying [on_____on____] to a neuron and spike train. Comparing models via neural predictivity: comparing two neural networks by their R^2 to mouse brain activity. RSA: assessing brain-brain or model-brain correspondence using representational dissimilarity matrices
View of London from a rooftop in Kings Cross
Panel discussion on the cognitive basis of reasoning in Minds and AI with Andrew Lampinen, Alison Gopnik, Laura Ruis, Andrew Granville, and Taylor Webb
Interventions can have opposite effects on behavior in an empty context and after a conversation
Relatively little representational change when contexts are sci-fi stories
Oscillating representations when a model plays both sides of a simulated argument
Representations in language models can change dramatically over a conversation. Conceptual overview: left is a stimulated conversation between a user and a model, right is a plot of the models linear representations of factuality of answers to questions like "do you have qualia" over the conversation — the answers that start factual flip over the conversation to non-factual, and vice versa.
Parametric cortical learning and hippocampal learning are complementary: parametric learning consolidates more experiences, but is more tied to learning formats/tasks, while episodic retrieval accesses fewer experiences but allows using latent information more flexibly
Our two methods for generalization may relate to studies on (respectively) offline and online hippocampal replay
In-context augmented fine tuning generalizes as well as ICL to reversals, better to syllogistic inferences.
In the reversal curse dataset (https://arxiv.org/abs/2309.12288) LMs finetuned on the forward statements do not generalize to the reversals — but ICL over the whole dataset gets 98% generalization.
Parametric learning in language models focuses on explicit structure, and loses latent information
Latent vs. explicit information in training data: 1) explicit forward relation vs. latent reversal, 2) multiple explicit statements and the latent syllogistic inference they imply, 3) explicit goals in a maze navigation task, and alternate goals encountered en route, 4) explicit statements in one language, and their implications for answering questions in another language
Title slide: distinct modes of generalization from parameters and context and two paths to bridge the gap
Wedding photo of us entering our reception
The benefits of oracle retrieval on the (a) Codebooks and (b) simple reversals benchmarks. Both baseline and retrieval models perform well on component tasks like recalling definitions, or encoding new sequences involving indices used in encoding during training (a, center). However, performance differs dramatically on the latent encoding test (right bars on both plots), where only the model with retrieval achieves above-chance performance.
The benchmarks we use and the key types of latent generalization that they test. (a) The codebooks benchmark tests the ability to use latent indices (highlighted in red) for which only the definitions have been seen in training to complete test encoding sequences. (b) The simple reversals benchmark tests the ability of models to reverse relations seen in training, and which models have learned to reverse in-context. (c) The semantic structure benchmark uses training embedded in more naturalistic text to test latent generalization types ranging from reversals to syllogisms, or more challenging category-inclusion-only holdouts. (d) The latent gridworld—with both its pixel-based RL and ASCII-based BC instantiations—tests the ability to navigate to objects that have never been a navigation goal in training for a particular maze, but have been frequently seen.
Explicit retrieval of learning experiences from nonparametric learning systems complements the broader knowledge of parametric learning—by making select, relevant experiences available in context where they can be more flexibly used in ways different from the original task setting in which they were encountered.
While a model may be trained on some explicit information (e.g. X is Y's teacher" or goals (e.g. navigate to Z), there may be other information latent in it (such as a reversal "Y is X's teacher). Challenges of reversal are one instance of the much broader phenomenon that what is explicitly learned may also latently convey information relevant to other tasks—e.g., multi-hop reasoning, alternative goals, or answering questions in other languages. Like the reversal curse, learning on such sequences may primarily improve performance on the explicit information or goals; however, if the sequence were in context, models would readily be able to make inferences about the latent information.
Homomorphic encryption: strongly dissociating computation from patterns of representation In the experiments described above, the role that the representations played in the computations of the system was relatively straightforward, even where the representations were biased. However, this does not have to be the case. We illustrate this with a final case study of the possibility for strong dissociation between computation and patterns of representation: homomorphic encryption (Gentry, 2009; Van Dijk et al., 2010). While the field of cryptography is largely focused on creating representations that preserve information yet are not easily decodable, in homomorphic encryption schemes it is additionally possible to perform arbitrary computations (any algebraic circuit) over this information while it is encrypted. That is, at each step of such a computation, a new encrypted representation is produced that corresponds to the result of encrypting the representation at that step of the original computation. This example shows that it is not necessary for a computational system to have any straightforward (e.g. linearly decodable) representation of the features that it uses in its computations. Systematic computations can be performed even over representations that are deliberately crafted to thwart attempts to understand (decrypt) their content. As a special case, this also illustrates that systematic compositional computations are possible without requiring representations that are straightforwardly compositional. Encrypted representations are compositional only in the sense that “with the right highly-nonlinear decoding scheme compositional representations can be extracted”—which is also true of some coding schemes typically interpreted as non-compositional, such as idiosyncratic representations of each input. This raises questions about if and when it is feasible to rigorously confirm whether a system’s computations are compositional from representational analyses.
Why are representations biased towards easier features? The biases are driven by multiple factors, including learning dynamics and the different ways that nonlinear features can be represented. (Left) By manipulating training order (training the hard task first rather than both simultaneously), the magnitude of the biases can be reduced. (Right) Likewise, by accounting for the fact that there can be more ways to represent a nonlinear feature that are not linearly equivalent—for example, different ways of drawing intermediate classification boundaries to compute an XOR function—we can identify other components of the representations that may be contributing to the model’s computation of the hard feature. Together the learning dynamics and multiple ways of representing features explain most of the representation bias towards the easy feature over the hard.
RSA within and between different sets of models can give surprising results due to representation biases. This plot shows similarities within and between different models computing different types of features. Ideally the similarities would be highest in blocks on the diagonal (i.e. models computing the same features), and the blocks off the diagonal would show graded similarity corresponding to the functional overlap. However, that is not the case. (Left) When comparing a model trained to output both easy and hard features to ones that are trained on only one feature, the multi-task model appears very similar to the easy-task only model (cf. Hermann and Lampinen, 2020). In fact, the models trained only on the hard task do not even appear particularly similar to other models trained on the same exact task. (Right) When models are trained on multiple easy or multiple hard tasks, the models trained on only hard tasks appear less similar to other models trained on exactly the same tasks than they do to models trained on strictly easier tasks that use the same input units.
Representational biases: in representations of a model computing easy (linear) and hard (4-parity) features, the overall variance explained in the last layer representations by the easy feature is over 55%, while the variance explained by the hard feature is around 5%. This is reflected in the top PCs clearly clustering by the easy feature but not reflecting the hard, and these biases are also present the unit level (almost all units, especially the most active ones), represent the easy feature more strongly.
Datasets where many input features (color, shape, texture, … size) are fed in to creating input data. Linear or nonlinear classification tasks can be created, e.g. classifying whether an object is a circle (linear) or whether it is XOR(yellow, checkered) which is nonlinear. Experiments: training neural networks to output multiple features computed from an input, e.g. a linear and nonlinear one. Learned representations: stimuli presented and datasets of representational activity from the model, as might be collected in a neuroscience experiment.
What do representations tell us about a system? Image of a mouse with a scope showing a vector of activity patterns, and a neural network with a vector of unit activity patterns Common analyses of neural representations: Encoding models (relating activity to task features) drawing of an arrow from a trace saying [on_____on____] to a neuron and spike train. Comparing models via neural predictivity: comparing two neural networks by their R^2 to mouse brain activity. RSA: assessing brain-brain or model-brain correspondence using representational dissimilarity matrices
Box 2 | Syntax & meaning in mathematics Newell & Simon motivated the physical symbol system hypothesis by this claim: “Logic, and by incorporation all of mathematics, was a game played with meaningless tokens according to certain purely syntactic rules. All meaning had been purged.” [1]. In the reasoning systems they created, symbols thus became synonymous with discrete entities, symbolic processing entailed manipulating with formal algebras, and the properties of our formal symbolic systems were inherited by GOFAI. But, is this how humans perform logical reasoning? Fields medalist Paul Cohen responded directly: “I never was able to successfully analyze proofs as a combinatorial ‘game’ played with symbols on paper,” instead, to reason productively “one must essentially forget that all proofs are eventually transcribed in this formal language” [77]. Mac Lane [78] explains why: “strict formalism can’t explain which of many formulas matter [...] the choice of form is determined by ideas and experience.” The meaning of symbols and their relationships gives mathematicians intuitions for why a theorem might be true, and thereby ideas of how to prove it (as do examples; 79). Furthermore, a major goal of theories and proofs is to convey understanding. Thus, mathematicians are often unsatisfied with a formal proof of a theorem that does not clearly communicate understanding of the underlying ideas [36, 80]. The meaning behind symbols is the essence of mathematics, not a burden to be discarded. McClelland et al. [65] draw on similar arguments to suggest that neural network models are uniquely well-posed to capture the meaning-laden processes of human mathematical reasoning
Across four dataset categories, ICL outperforms finetuning, and augmented finetuning outperforms both, though gaps are generally smaller than in the simpler experiments. Interestingly, finetuning performs above chance in some conditions like reversals. Category splits remain difficult for all methods
Syllogisms: finetuned models perform at 35% (somewhat better than chance), ICL performs around 60%, augmented finetuning (see below) performs around 83%
Reversal curse: pretrained models perform at chance on the data, finetuned models perform at 0% (worse than chance), ICL performs around 97%, augmented finetuning (see below) performs slightly better.
Overview: If we have a language model and a new dataset, we can incorporate it in two ways: either finetuning the model on the dataset, or just putting the dataset in the context of the original model. Which generalizes better to held out test questions?
Plot of train and test accuracy vs. spurious feature predictivity in training. Train accuracy is always 100%. Test accuracy is 50% if the spurious feature is unpredictive, or 100% predictive. But there is a inverted-u-shaped benefit where starting from around 50% training predictivity of the spurious feature, and peaking around 75% training predictivity, the model achieves near-perfect generalization to the test set, even though the spurious feature is always unpredictive on the test set.
Could spurious features improve transformer generalization? Train: sentenes have syntax (hard, but 100% predictive of the label) and content (easy but 75% predictive). But content gives a hint about the label. Test: under distribution shift; syntax is still 100% predictive of the label, but content is useless.
Two ways of processing the sentence "The dog that the people on the corner own barks at the car.” 1) a syntactic tree (a hard feature) 2) the content: It’s usually dogs that bark at vehicles. It’s usually people that own animals. (2) gives a hint about (1).
Figure 7: Overview of how we can develop theories with potentially opaque models and unnatural manipulations of natural data.
Two ways of deconfounding a natural distribution: use a simplified artificial one (narrow), or systematically manipulate the broader distribution to deconfound it (while maintaining variability).
Overview of key strategies cognitive science can adopt to develop generalizable models: frictionless reproducibility, seeking external validity, and building dynamic phenomena-oriented benchmarks
Overviews of the benefits of increasingly naturalistic experimental conditions and learning with naturalistic data for understanding the origins and function of cognitive capabilities
Naturalistic computational cognitive science: the what, why, and the how. What: Section 2: Develop learning-based models of intelligence that can predict human behavior on both simplified and naturalistic stimuli and tasks Why: Section 3: Why increase the naturalism of our experimental paradigms? Section 4: Why learn models of intelligence? How: Section 5: How do we build generalizable models that scale to naturalistic settings? Section 6: How do we build and test theories with naturalistic tasks and models?
All three models show stronger shape bias when it is mentioned first in text than when color is mentioned first, though the magnitude of the effect varies across models and tasks. (For the single-category generalization task, Claude & GPT refuse all generalizations for these text descriptions.)
Across three models (Gemini 1.5 Pro, Claude 3.5 Sonnet, and GPT4o mini), there are strong effects of presentation modality on how the models generalize. The models generally show higher shape bias in images than in text, though for the odd-one-out task Claude and GPT surprisingly show. the opposite effect.
Figure showing experimental paradigms and evaluating different modalities and formats: Our paradigms include: One-category generalization, where multiple examples of e.g. a green cube are presented, and then the model is asked whether a yellow cube is a dax, and whether a green cylinder is a dax. Two-category cue-conflict, where the model is told a green cube is a "dax" and a yellow cylinder is a "fep", and then is asked to choose which category a green cylinder belongs to, and which category a yellow cube belongs to. Odd-one-out, where the model is shown several green cubes, a yellow cube, and a green cylinder, and is asked which is the odd one out. Modalities: We experiment with presenting the examples as images, or as text descriptions, and with different ways of phrasing the text descriptions that shift the feature order (e.g. "an object that has shape cube and color green").
Felix Hill and some other DMers and I after cold water swimming at Parliament Hill Lido a few years ago
Paper screenshot: Nevertheless, the association with formal education raises an intriguing question: could language models learn to reason more reliably with targeted formal education? Several recent results suggest that this may indeed be a promising direction. Pretraining on synthetic logical reasoning tasks can improve model performance on reasoning and mathematics problems (78). In some cases language models can either be prompted or can learn to verify, correct, or debias their own outputs (79). Finally, language model reasoning can be bootstrapped through iterated fine-tuning on successful instances (80). These results suggest the possibility that a model trained with instructions to perform logical reasoning, and to check and correct the results of its work, might move closer to the logical reasoning capabilities of formally educated humans. Perhaps logical reasoning is a graded competency that is supported by a range of different environmental and educational factors (28), rather than a core ability that must be built in to an intelligent system.
Given a task to be learned, such as the function f(x) = 2(x + 3), there are many ways that task could be presented in context — not just few-shot examples, but also instructions, or a graph. Likewise, there are many ways that the knowledge could be tested or applied once learned; e.g. asking the model to write code implementing the function, or give its inverse, etc.
Two examples of linguistic dependencies: “Alice does power lifting. Bob does ballet. She goes to the gym. He goes to the studio.” The parallel structures and coreference (e.g. she -> Alice) are similar to a few-shot supervised ICL task. “The people at the adjacent table listen.” The syntactic dependency (listen -> people) is a simple kind of context adaptation.
A Venn diagram showing ICL as a broad category, of which few-shot ICL is a subset. Other sub-categories include instruction following, and even some types of more basic linguistic structure like coreference or parallel structures.
Language models do many other kinds of in-context adaptation, which don’t fit cleanly into the standard few-shot supervised paradigm Examples of learning tasks from instructions (“translate English to French”), role play “The masterful French translator flawlessly translates the sentence into English”, random or missing labels, or shots with explanations.
An outer loop of pretraining, across a broad distribution of many documents with mixtures of contextual properties that recur (repeated chunks, academic formats, syntactic structures, etc.) yields emergent in-context learning: adaptation to properties within a context, yielding lower loss towards the end.
Aligned model performance improves on ML tasks: 1-shot performance generally improves across a range of datasets including birds, places, UC Merced, etc. Distribution shift robustness improves notably on the BREEDS dataset, as does ImageNet-A accuracy.