Alex Crenshaw

@crenshaw.bsky.social

282 Followers 713 Following

Assistant Professor of Psychology. Couples, clinical trials, stats & methods. Views my own.

1. Report how the mean difference was computed 2. Report the exact SD used. Citing a paper isn't enough b/c our review found that authors use terms like “Cohen’s d” or popular citations like Feingold (2009) to indicate different methods. 3. Use the term "SMD" instead of terms like Cohen's d, which no longer imply a specific formula in modern usage. Then report the exact operationalization 4. Prespecify the operationalization 5. Always report the mean difference in original units alongside the SMD
Table 1: Desirable properties of the SMD and extent to which each standardizer meets each property
Figure 1: Sankey diagram showing broad and specific operationalizations of the SMD in reviewed clinical trials
Screenshot of text. Text reads: Finally, Luo et al. (2022) conducted perhaps the most thorough analysis by computing the SMD in three or more distinct ways for each individual trial that reported sufficient information to do so, finding the median difference in magnitude between the largest and smallest SMD was 0.30 (range = 0.02–1.61). This value is comparable to proposed values for the smallest worthwhile differences for psychotherapy (e.g., Sahker et al., 2025, Supplement 5). Therefore, individual researchers’ choice of standardizer alone, holding all other factors constant, has the potential to influence clinical decision making.
Abstract: The standardized mean difference (SMD) is widely used in psychology clinical trials to convert intervention effect estimates from raw units to a common scale, enabling comparison across settings. However, multiple operationalizations of the SMD are possible for clinical trial data, with large potential impacts on estimate magnitudes. In this commentary, I review practices for operationalizing the SMD in the Journal of Consulting and Clinical Psychology and other journals that publish results of clinical trials. I find that only one third of studies report sufficient information to determine the operationalization. Among those, the operationalization varies widely, threatening the SMD’s utility in providing a common scale. Given the common goals and design elements across many trials, it is feasible to establish a default operationalization for the SMD’s standardizer. I outline eight desirable SMD properties and evaluate each option against these properties. The pooled baseline standard deviation meets seven of eight, far more than any alternative, with its main limitation being susceptibility to inflation from restrictive inclusion criteria. I conclude that raw effect estimates should be divided by the pooled baseline standard deviation as a default, one-size-fits-most choice. Adopting this standard will help the SMD meet its promise as a standardized metric, improving comparability of results across different measures, outcomes, studies, and populations. I also provide reporting recommendations to improve transparency and enable recomputation using alternative standardizers when needed.
1. Report how the mean difference was computed 2. Report the exact SD used. Citing a paper isn't enough b/c our review found that authors use terms like “Cohen’s d” or popular citations like Feingold (2009) to indicate different methods. 3. Use the term "SMD" instead of terms like Cohen's d, which no longer imply a specific formula in modern usage. Then report the exact operationalization 4. Prespecify the operationalization 5. Always report the mean difference in original units alongside the SMD
Table showing how well each operationalization meets desired SMD properties
Sankey diagram showing broad & specific operationalization of SMDs reported in clinical psychology trials
Finally, Luo et al. (2022) conducted perhaps the most thorough analysis by computing the SMD in three or more distinct ways for each individual trial that reported sufficient information to do so, finding the median difference in magnitude between the largest and smallest SMD was 0.30 (range = 0.02–1.61). This value is comparable to proposed values for the smallest worthwhile differences for psychotherapy (e.g., Sahker et al., 2025, Supplement 5). Therefore, individual researchers’ choice of standardizer alone, holding all other factors constant, has the potential to influence clinical decision making.
Figure showing multiverse results for PTSD symptoms during treatment phase
Table 2 from manuscript detailing multiverse dimensions
Table 1 from manuscript detailing multiverse terminology, expanded decision types from Del Giudice & Gangestad (2021), and decision-making guidelines
Title page and abstract. Abstract text: "Making decisions regarding data processing and analysis are crucial steps toward extracting insights from data in clinical trials. Trial registries like clinicaltrials.gov promote transparency about these decisions and encourage making them in advance. However, clinical studies often face decisions with multiple reasonable options outside the bounds of preregistration, such as when studies conduct post hoc analyses, deviate from preregistered plans, or simply were not preregistered. Additionally, even a priori decisions often have multiple reasonable options from which to choose. Methods that maximize transparency and minimize bias in such situations are needed. This paper advocates for applying a “multiverse” approach to analyzing such data from clinical trials. The multiverse approach simultaneously selects and analyzes the various reasonable options for each decision and presents results across all analysis “universes.” We highlight common challenges and decisions when analyzing clinical trial data, review and expand upon the multiverse approach and show how it can address these challenges, and demonstrate the approach using data from a small randomized psychotherapy trial for posttraumatic stress disorder. In the example presented, results were fully consistent across the multiverse for one outcome (posttraumatic stress symptoms), partially consistent for another (relationship satisfaction), and mostly inconsistent for a third outcome (fear of intimacy). The multiverse approach is a flexible and transparent analysis option for clinical trials in the presence of uncertainty regarding data processing and analytic choices."
Detailed Recommendations for computing the reliable change Index: (1) Use a premade RCI threshold if one has been previously developed from a large sample of a comparable population. (2) If a premade RCI threshold is not available or there is a substantial population mismatch between the premade threshold and one’s study, (a) If N > 100, use the sample estimates of s and rxx to compute the RCI. (b) If N < 30, use estimates of s and rxx from a large-N study that best matches one’s population of study. Measure validation studies are a good default choice. (c) If 30 ≤ N ≤ 100, make a determination based on the study sample size and availability and representativeness of thresholds from a larger sample. (3) Do not use estimates of s and rxx from different samples. (4) Justify the choice of method for the RCI and its specific operationalization, and discuss limitations and biases that may result. Document such choices ahead of time in a study preregistration due to the impact that these choices may have on the strength of apparent evidence for change in a study.
Figure 1. Percent of samples with [y] or greater deviation from the population value for the reliable change Index: simulated data. The y-axis represents the percent absolute deviation in the Reliable Change Index, and the x-axis represents the percent of samples that have a deviation as large or larger than the y-axis value. Text labels are provided for the worst 10% and 20% of samples, and for the median sample (50%). Example to aid interpretation: for the high reliability/more items measure (top left panel) when N = 10, 10% of samples (x-axis) have a deviation from the population value of 13% (y-axis) or greater, 20% of samples have a deviation of 10% or greater, and 50% of the samples have a deviation of 5% or greater.
Title page and abstract. Abstract text: "The reliable change index (RCI) is a tool for evaluating change at the individual level. It compliments standard group-level change estimates and is central to evaluating clinical significance for interventions. In principle, the RCI provides common criteria for evaluating individual change. However, current practices use sample-specific estimates to create these criteria. Because sample estimates are subject to sampling error, these criteria are also subject to sampling error and therefore differ across studies. We illustrate how current practices can lead to differing criteria for reliable change and use simulations to identify the impact of sampling error on the RCI. Excessive error in the RCI began for the average sample when N < 30, and samples only comfortably avoided the risk of excessive error when N > 100. Finally, small errors in estimating a measure’s reliability sometimes had profound effects on the RCI. Recommendations on use of the RCI are provided."