Martin Modrák

@modrakm.bsky.social

1339 Followers 46 Following

Biostatistics/bioinformatics at Charles University, 2nd faculty of Medicine. Bayesian in practice, but not a fan of Bayesian epistemology. Main on fedi: https://bayes.club/@modrak_m Blog: https://martinmodrak.cz

Title: What’s a multiverse good for anyway? By: Julia M. Rohrer, Jessica Hullman, and Andrew Gelman Abstract: Multiverse analysis has become a fairly popular approach, as indicated by the present special issue on the matter. Here, we take one step back and ask why one would conduct a multiverse analysis in the first place. We discuss various ways in which a multiverse may be employed – as a tool for reflection and critique, as a persuasive tool, as a serious inferential tool – as well as potential problems that arise depending on the specific purpose. For example, it fails as a persuasive tool when researchers disagree about which variations should be included in the analysis, and it fails as a serious inferential tool when the included analyses do not target a coherent estimand. Then, we take yet another step back and ask what the multiverse discourse has been good for and whether any broader lessons can be drawn. Ultimately, we conclude that the multiverse does remain a valuable tool; however, we urge against taking it too seriously.
Abstract: I argue that multiverse analysis is highly suited to two undesirable uses: abdication of researcher's responsibility for their conclusion and manufacturing of doubt. A review of multiverse analyses published in 2025 provides tentative empirical support that abdication of responsibility is present in the literature and I mention anecdotal evidence that multiverse has been used for manufacturing of doubt about Covid-19 precautions. To mitigate negative effects if multiverse analysis becomes widely used I suggest the community adopts two conventions for evaluating multiverse analyzes: evaluating multiverses by the single worst universe they contain and considering large size of a multiverse as a sign of weakness rather than a praiseworthy achievement.
Quote from the ACOUP blog: How does this error happen? It tends to come from two main sources. First, it usually occurs most dramatically in military systems where the military leadership – which has been trained for operations and tactics, not strategy, which you will recall is the province of kings, ministers and presidents – usurps the leadership of the state. Second, it tends to occur when those military leaders – influenced by their operational training – take the operational conditions of their planning as assumed constants. “What do we do if we go to war with the United States” becomes “What do we do when we go to war with the United States” which elides out the strategic question “should we go to war with the United States?” entirely – and catastrophically, as for Imperial Japan, the answer to that unasked question of should we do this was clearly Oh my, NO. (Bibliography note: It would hardly be fitting for me to declare these errors common and not provide examples. Two of the best case-studies I have read in this kind of strategic-thinking-failure-as-organizational-culture-failure are I. Hull, Absolute Destruction: Military Culture and the Practices of War in Imperial Germany (2005) and Parshall and Tully, Shattered Sword: The Untold Story of the Battle of Midway (2005). Also worth checking out, Daddis, “Chasing the Austerlitz Ideal: The Enduring Quest for Decisive Battle” in Armed Forces Journal (2006): 38-41. The same themes naturally come up in Daddis, Withdrawal: Reassessing America’s Final Years in Vietnam (2017)).
Photo of me, holding a copy of the Bayesian Workflow book, smiling like a happy statistician fella that I am.
A screenshot of the front page of Cross Validated, showing that all activity in the past 7 hours is the "Community" bot bumping up old unresolved questions.
A plot of histories of check statistics for a Bayes factor computation with bad normalization constant. Values below the blue horizontal line indicate a problem is signalled. It shows that SBC for the `model` and `sigma` parameters discovers the problems on average after ~270 and ~ 200 simulations respectively, binary calibration and data-averaged posterior (DAP) needs just ~150 simulations.
A plot showing perfect binary calibration of a prediction model.
Making research data and other intermediate research outputs more freely available should accelerate the production of knowledge and improve quality through error detection (both of which are demonstrably at least partly true). But now we are seeing readily available data assets, such as GWAS summary statistics and observational cohort study data from the UK Biobank, NHANES, and many other databases, being weaponized to generate manuscripts with minimal scholarly value and utility (or even negative utility given the burden they place on the system). In many ways, the story of MR recapitulated what had happened previously with meta-analysis. The aggregation of existing data from combinable randomized trials led to many important clinical and public health advances, and to less research waste and better science going forward. However, the combination of using data that could be extracted from existing publications with relatively simple statistical analyses led to an explosion of publications of decreasing and then negative scientific value. Thus, despite best intentions, efforts to improve research rigor, such as by making data available, synthesizing evidence, and improving causal inference methods, have become commodified by authors and paper mills to increase the marketability of a manuscript, rather than to improve the quality of scholarship.
An RStudio console with output of a Stan run. It ends in 3 warnings: 1: There were 38 divergent transitions after warmup. 2: There were 836 transitions after warmup that exceeded the maximum treedepth. 3: Examine the pairs() plot to diagnose sampling problems
A plot of coverages of 95% intervals of brms, glmmTMB, gamlss and glm.nb for a between-group difference in negative binomial model. Brms attains good coverage, but other approaches result in too low coverage across the board.