Matthew B. Jané

@matthewbjane.bsky.social

5275 Followers 325 Following

PhD student at UConn | matthewbjane.com

Screenshot of a book cover: “Doctored: fraud arrogance, and tragedy in the quest to cure Alzheimer’s” - by Charles Piller
From footnote 3 of R&H’s article: Ferguson carried out a “random effects model meta-analysis” that is meant to approximate the size of a single effect based on data from a collection of experiments measuring this single effect. For such a meta-analysis to make sense, the 'effect' must be well-defined, and all the experiments must measure the same effect — at least practically the same effect. Furthermore, the word 'random' indicates the presumption that the variation in experimental outcomes is indiscernible from the influence of a mere chance. As we will see, these requirements for a valid 'random effects model meta-analysis' are all violated. In addition, we will also see that Ferguson fails to conduct some basic elements of a 'meta-analytical review' (study of studies).
Livestream thumbnail with the title “Coffee, Lo-Fi, and Statistics”
Left side has a forest plot with the distribution of true effect sizes on top. The following things are labelled: mean of true effect sizes (mu), standard deviation of true effect sizes (tau), true effect sizes (theta), 95% prediction interval, sampling error (epsilon), sampling variance (v), observed effect sizes (h), estimate of the mean of true effect sizes (mu hat), 95% CI of mean of true effect sizes. Right side of figure has the model for effect sizes h = theta + epsilon. It also shows the distributions of true effect sizes theta ~ Normal(mu, tau^2) and the conditional distribution of observed effect sizes (i.e., the sampling distribution) h | theta ~ Normal(theta, v).
To the left is a rectangle representing the sample space. The dots are the elements of the sample space, each dot has a line that connects the sample space to the real line. There are many lines that connect to the real line forming a distribution described by some density function f_V(v)
Two plots showing pre/post change. Plot to the left shows pre/post change in the control group, and the right plot shows pre/post change in the intervention/treatment group
Two crew neck sweaters, a backpack, a thermos and business card all with my logo on it
Diagram showing X -> S(Z) -> X|S(X) showing the selection process
Your comment is awaiting moderation. Yes there is. So if you know the average inter-trial correlations and you know the number of trials. You can obtain the reliability via spearman-brown prophecy formula: rXX = k*r_trials / (1 + (k – 1)*r_trials) Using the estimated reliability, you can correct for bias by dividing the standardized mean difference by the square root of the reliabilities in X and Y For Cohen’s d: d(corrected) = d / sqrt(rYY) For Pearson Correlation: r(corrected) = r / sqrt(rXX*rYY) The correction will also need to be applied to the standard error as well. You can find these equations in Hunter and Schmidt’s textbook “meta-analysis methods” or in the fantastic paper by Brenton Wiernik: https://journals.sagepub.com/doi/full/10.1177/2515245919885611
Normal distribution colored red on the lower half, and blue on the upper half
Bivariate normal scatter plot. Both variables are continuous. The top corner of the plot shows a correlation coefficient of .50
Plot showing sample size on X-axis and correlation on Y-axis. The plot is a heat map with darker colors showing more bias when the correlation is around .5 and the sample size is small
Plot showing how R^2 averaged over many samples (y-axis) will show substantial effect sizes in low sample sizes (x-axis)