Paweł Lenartowicz

@freestylerscientist.pl

478 Followers 1138 Following

Aspiring statistician and meta-researcher, sailor and activists. #PhilosophyofScience #OpenScience #Uncertanity

So, here you are, I replicate exactly the same problem with empirical data from the FIRST open source article I found. (posting my browser history is sufficient?) > power [1] 0.6367 > p_beetween [1] 0.1702 Source of data: https://www.nature.com/articles/s41467-024-55155-4/figures/1 (excel in attached)
Another issue, which to my knowledge has not been widely discussed, is that p-curves — and consequently the z-values of some tests based on the t-distribution — do not always follow their theoretical distribution. This discrepancy arises when a t-test is applied to a heavy-tailed distribution, where individual data points can disproportionately impact the standard deviation estimate. In such cases, the distribution of t-statistics can shift toward values around 1 or -1. This characteristic, while making the t-test robust and conservative under assumption violations by preventing inflation of the error rate beyond the nominal $\alpha$-level (at the cost of statistical power), complicates interpretation near the significance threshold.
nSims <- 100000 #number of simulated experiments p <-numeric(nSims) #set up empty container for all simulated p-values n=20 simulation_chi_selective <- function(effect, n, ratio, df){ m = rbinom(1, n, ratio ) s2a = rnorm(n-m) s2b = rnorm(m) + rchisq(m, df = df) * effect s2 = append(s2a,s2b) return(s2) } for(i in 1:nSims){ #for each simulated experiment x<-rnorm(n = n, mean = 0, sd = 1) y<-simulation_chi_selective(effect = 2, n = n, ratio = 0.25, df = 8) z<-t.test(x,y) #perform the t-test p[i]<-z$p.value #get the p-value and store it } #now plot the histogram hist(p, main="Histogram of p-values", xlab=("Observed p-value"), breaks=40) #Observed Power power <- p[p<0.05] power <- length(power)/nSims p_beetween <- p[p<0.05 & p>0.025] p_beetween <- length(p_beetween)/nSims
"blue cabbage" or modra kapusta on picture.
Text show correction changing "questionable" to "different" research practices.
The null ritual: 1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research hypothesis or of any alternative substantive hypotheses. 2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as p < 0.05, p < 0.01, or p < 0.001 (whichever comes next to the obtained p-value). 3. Always perform this procedure.