Ryan Briggs

@ryancbriggs.net

5451 Followers 1058 Following

Raising kids & bread & grant money. Cleaning data & diapers & fish. EA (bed nets, not light cone). Social scientist. typos. twitter.com/ryancbriggs

A photo of Gosset in the Guinness Storehouse. The quote reads “IN HIS WORK WITH BARLEY RESEARCH, GOSSET DREW CONCLUSIONS FROM ANALYSING SMALL SAMPLE SIZES....”
A large bowl of shaved ice (plain and mango), along wild bowls of taro balls, panda jelly, peaches, and evaporated milk.
A cherry clafoutis
1. Course Introduction and Setup
 Course overview; installing RStudio; introduction to causal inference; ModernDive Chapters 1–2 for newcomers. 2. Data, Tidy Data, Wrangling, and Visualization
 Core R skills for importing, cleaning, reshaping, summarizing, and visualizing evaluation data. 3. Sampling, Uncertainty, and Inference 
Sampling variation, confidence intervals, hypothesis testing, and the logic of statistical uncertainty. 4. Difference in Means as Regression 
Equivalence between difference-in-means estimates and lm(y ~ treat); ATE as the treatment coefficient; control mean as the intercept; covariates for precision gains; simulations and re-analysis of Karlan–List charity data. 5. Interactions and Treatment Effect Heterogeneity 
Interaction terms, subgroup analysis, heterogeneous effects; simulations, Karlan–List charity data, and Thornton HIV data. 6. Standard Errors, Power, and Research Design 
Bias, variance, RMSE, clustering, power analysis, and how underpowered studies contribute to selection on significance and inflated estimates. 7. Noncompliance, Take-Up, and Instrumental Variables 
ITT, TOT, LATE, compliers etc, and randomized encouragement designs; Thornton HIV testing incentives; reading from The Effect Chapter 19 or Causal Inference: The Mixtape IV chapter. 8. Spillovers, Externalities, and Peer Effects
 How spillovers can bias experimental estimates; identifying, measuring, and interpreting spillover effects in development evaluations. 9. Pre-Analysis Plans, Measurement, and Cost-Effectiveness
 PAPs, outcome measurement, measurement error, index construction, and basic cost-effectiveness analysis. 10. Meta-Analysis and Evidence Aggregation 
Fixed-effect and random-effects meta-analysis; Bayesian meta-analysis using baggr; interpreting accumulated evidence across studies. 11. Case Study: Deworming Evidence I
 Critical re-analysis of the main deworming results; statistical interpretation; cost-effectiveness implications.
7 some tests should not be published So far, our discussion of SoS has treated hypothesis tests as if they all deserved to be published, regardless of how they were generated. That simplification is useful for diagnosing the problem, but it is not a sensible norm for research practice. In fact, many results are not worth the time and attention required to write up, review, and archive them in the permanent scientific record. Estimates can be uninformative, poorly motivated, or produced by a broken design. In such cases, shelving them is not a distortion of the record so much as basic quality control. Given limited journal space, reviewer resources, and reader attention, some results will inevitably be filtered out of the literature. The key point that we want to emphasize here is that most of the good reasons we have to choose which tests to publish apply to both null and significant results. In this section, we consider three broad categories of reasons to filter out tests: quality, precision, and dosage. Crucially, of these three categories, only the last applies specifically to null results.
We can now address the question that many researchers implicitly pose when faced with a null result: does a small |𝑧| constitute evidence of failed delivery? The intuition that motivates this question is partly correct. Indeed, a large |𝑧| in a plausible direction does shift posterior mass toward higher 𝑑, because Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) assigns more probability to extreme primary statistics when dose is high. However, two structural features limit how far this update goes. Consider the posterior probability that the dose was properly administered, when we have no auxiliary evidence. That probability is proportional to the distribution of the test statistic 𝑧, weighted by the prior probability of each dose level: Pr(𝑑 ∣ 𝑧) ∝ Pr(𝑧 ∣ 𝐷 = 𝑑) ⋅ Pr(𝑑). The challenge is that, since the treatment effect 𝑇 is unknown, the likelihood of 𝑧 is a mixture of two components: with and without a true effect. Pr(𝑧 ∣ 𝐷 = 𝑑) = Pr(𝑇 = 1) ⋅ Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) + Pr(𝑇 = 0) ⋅ Pr(𝑧 ∣ 𝑇 = 0). Neither component is very helpful for pinning down 𝑑. Under 𝑇 = 1, the likelihood does depend on 𝑑, but only through the same product ambiguity noted above: any combination of effect size and delivery yielding the same product is observationally equivalent. Under 𝑇 = 0, the likelihood does not depend on 𝑑 at all, so it contributes nothing to dose inference. As a result, even extreme 𝑧 statistics can lead to only modest updating about 𝑑. The intuition that “a null result implies weak delivery” is not wrong per se, but it is prior-sensitive and noisy.
A table showing abstract shares of null and non-null results split over many features of articles. Field Experiments have relatively high null reporting.
A graph showing how expected null only abstract shares vary with the number of tests conducted and the per-test probability of a null result.
6 chocolate chip cookies cooling on a rack
If we see falling costs across many factors of research production, with the effect that humans start increasingly focusing their effort on the remaining stubborn tasks, then we should see overall research output rise. If this occurs then it seems quite likely that in short order the true bottleneck on the creation of socially valuable knowledge will not be in research production at all. Instead, the bottleneck will be in research consumption. In this world, I expect two constraints will start to bind. The first is adjudication capacity, or simply figuring out which claims are worth trusting. One approach to this is to read each paper carefully, but everyone wants to defer at least some of this task to someone else. The second is the task of filtering research according to various criteria such as importance. We currently use the peer reviewed journal system for both of these tasks.
Information manipulation is going to get cheaper much faster than interacting with the world will. I expect that this will lead to the bottleneck in research shifting from production to consumption. We should be preparing for this shift now by building infrastructure that allows us to adjudicate claims and filter work in a world where production is cheap. The current journal system is not well positioned to do this, and so we should be experimenting with new institutions that can.
A note to Santa asking for cash transfers
A screenshot showing: Introduction These are notes for my class on probability models. In these notes, I walk through the concepts and computation that support modern probability modeling in political science using both maximum likelihood and Bayesian approaches. The Goal There are many excellent books on probability models. But I felt the need to write my own. Why? I saw three problems. First, some classes assign a huge textbook. It might be possible for the strongest and most motivated students to become familiar with the range of topics covered in these textbook, but impossible to master. Instead, these textbooks seem like references, something you’re supposed to constantly be referring back to throughout your career. I know this because many of these books have instructors’ guides that suggest what should be covered in a single semester, what should be skipped, and how one might jump around. Instead, I want a book that students can work through beginning to end and master each idea. Second, some classes assign a variety of sections from several books and a collection of articles. But then the story told in the readings isn’t coherent. The styles are changing, the author’s tastes are changing, and the notation is changing. Switching among authors can feel like whiplash when learning a difficult subject. Instead, I want a book that tells a continuous story with consistent style, tastes, and notation. Third, some classes assign readings that support the lecture material, without exact alignment between the two. For better or worse, the content covered by the instructor in class feels like the most important material. Thus, I want a book that exactly aligns with the material I cover in class.
Homemade scone with passionfruit curd on one half and clotted cream and raspberry jam on the other.
Abstract of a paper by Don Knuth claiming that Opus 4.6 solved an open problem he had been working on for weeks.
Ryan and 3 kids working with dough
a Galton board that shows selection on significance
A Galton board that shows 1 tailed selection on significance
An illustration of selection on significance for a test with 50% power
example podcast transcription
a graph showing how LLMs do or do not p-hack
Holistic Causal Learning with Causal Graphs: A Credible Method for Study Design and Preregistration in the Social Sciences While research designs in the social sciences have employed increasingly sophisticated methods to control false positive rates, there is still substantial debate about the merit of pre-registration and other recent open science reforms. In this paper, I present a method for preregistering causal graphs and employing the metric of entropy, and in particular Jaynes’ theory of maximum entropy, to propose a holistic way of measuring the contribution of a research study. To demonstrate the method’s utility, I show how recent research in both COVID-19 and political authoritarianism can be fruitfully understood using causal graphs and entropy. Additionally, I provide R code to enable researchers to compute these metrics, helping them prepare for various outcomes and learning approaches for the purpose of preregistration.
how the article share of various methods is changing over time
Graphs of sensitivity, showing LLMs outperforming humans
a graph showing JEPS has less selection on significance than other journals
a table of numerical results, as described in the post
It must be very hard to publish null results Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100,000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science.
"There is only one test" schematic image
Pandan crème brûlée in a mason jar with a small spoon.
Maple pecan slice-and-bake cookies on a silpat
a light table showing 15 slides covering poverty statistics and a photo of a baby. one slide reads "DON'T BE NIHILISTS"
Learn dual use skills During your PhD, learn skills that are rewarded both in the academic and non-academic job markets Programming and causal inference Languages and cultural competence Ideally, also learn skills that make you an expensive complement to something that is getting cheaper Data science
Fast job market advice 1. Realize that the job market is a signalling game 2. Have a model 3.Playing it safe is not safe 4. Learn “dual use” skills
brief description of my experience and a joke xkcd about selection bias
A seasoned steel wok
a galton board
A chocolate sesame cake, cooling on a rack
Excerpt from Cohen's textbook that reads "It is proposed here as a convention that, when the investigator has no other basis for setting the desired power value, the value .80 be used. This means that b is set at .20. This arbitrary but reasonable value is offered for several reasons (Cohen, 1965, pp. 98-99). The chief among them takes into consideration the implicit convention for a of .05. The b of .20 is chosen with the idea that the general relative seriousness of these two kinds of errors is of the order of .20/.05"
a graph showing that power increases most steeply per SE reduction around 80%, before declining quickly
Title: A Justification for 80% Power Abstract: Cohen’s heuristic reason for choosing 80% power (balancing Type I and TypeII errors) conveniently arrives at approximately the same number as an approachwhere one maximizes the marginal gain in power per standard error reduction. Ihave yet to see someone point this out, and this is interesting because it providesa non-arbitrary justification for 80% power.
Testing for Underpowered Literatures
Do Better Journals Publish Better Estimates?
Testing for and Evaluating the Extent of Selective Reporting
Andrew Gelman writing a list of wrong things on a chalkboard
Children’s art that says “happy Thanksgiving. I am thankful for toast.”
Two round apple challahs on a baking sheet