Lisa DeBruine

@debruine.bsky.social

7098 Followers 405 Following

#metacheck #rstats #PsyTeachR #OpenResearch #ManyFaces #iyknyd (overwhelmed by social media) 🏳️‍🌈 she/they

A person with blue hair and a rainbow scarf and rainbow face jewels.
An outdoor concrete staircase photographed from a low angle.
A pair of centaur parents in the distance, looking at three centaur babies. The horse part is galloping, while the human part is a floppy baby.
Inclusive pride flag flying over the University of Glasgow Gilbert Scott building.
Ingest research materials in structured, systematic manner Define dimensions along which to compare the research objects Extract text from each object which best relates to a given dimension of comparison Adjudicate on whether the extracted text matches between research objects
Why not just throw this at an LLM? * They are very sensitive to different prompt features, input parameters, etc. (Cummins, 2026; Sclar et al., 2023) - we need standardized inputs * They will likely be accessed via commercial chat interfaces, which are not appropriate for robust research activities (Cummins et al., 2025) * Users will use different models and we will have little control over this * Data privacy will likely be an afterthought * Asking an LLM to do comparisons all at once (e.g., sample size, hypothesis tests) necessarily introduces dependencies between these, which makes reproducibility and meaningtul evaluation harder to achieve * All of the above can be resolved by using a bespoke, modular workflow
A Conceptual and Technical Framework for Automating Research Trustworthiness Assessment Dr Jamie Cummins
A bunch of yellow products ona yellow background with simple names.
Postcard with a picture of a chihuahua reading: Register your O’Reilly Book!
Results We were unable to identify any randomised controlled trials of parachute intervention. Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.
Graph of new submissions by month from 2017 to 2026. Submission increased a bit in 2020 and doubled again in 2025.
Lisa next to a blue R and a small stuffed bear with a JASP t-shirt sitting on top of a pile of books (Queer Data, The 7 Deadly Sins of Psychology, Invisible Women, How to Design and Report Experiments, Linux in a Nutshell, Data Analysis Using Regression and Multilevel/Hierarchical Models, and Hays Statistics)
Results and discussion This part show how change in number make line in graph up or down for speed, spin, hot, and mix in layer. Solve in MATLAB BVP4C and get line and table for skin friction, Nusselt, Sherwood. Magnetic M make speed small, spin more near wall, and hot more because electric hot. Micro N change spin and speed near sheet. Light heat R make hot layer thick and more hot in flow. Soret Sr make mix from hot change, Dufour Du make hot from mix change and less near wall. Chemical Kr eat mix near sheet. Ohmic hot from M and σ put more hot inside, viscous warm Ec make more hot from speed change. When hole in medium big (φ), speed go small, hot and mix go more. Permeability K change how easy flow go in hole. All show in figure line go up down, use in filter, oil take, heat machine, and many place in work.
This work look for Soret and Dufour thing on MHD flow for micropolar fluid on sheet with hole stretch, also see hot from electric and warm from move, which old study no much talk. These hot and mix change speed, hot, spin, and mix of fluid in layer. The big math changes to small math by same change and solve in MATLAB BVP4C. Picture line and table show number for skin friction, Nusselt, Sherwood. We see more M make slow speed, light heat make thick hot layer, more micro make more spin near wall. Also, hot from electric and warm from move make more hot, so need in work like earth heat take, oil get, and hot change machine in hole thing.
I scored 11% all in the disruption category.
Table showing: Increasing usefulness (and effort) [Database logo] 1. Curated data available in a findable source (e.g., OSF) 2. Curated data permanently archived in a reputable archive [Code logo] 1. Workflow documented in text or video for exact reproduction 2. Workflow coded or scripted for automatic reproduction 3. Code/scripts available in a findable source (e.g., GitHub) 4. Code/scripts permanently archived in a reputable archive [Software blocks logo] 1. Software and dependencies mentioned in a README 2. Specific versions of software and dependencies are mentioned 3. Software and dependencies provided or linked in permanent archive 4. Exact software and dependencies provided in a container [Computer logo] 1. Computational environment (e.g., operating system) mentioned in a README 2. Exact computational environment provided in a container [Hardware logo] 1. Hardware requirements described in a README 2. Choose to use specific hardware as agreed in your community
XKCD comic: Title: HOW STANDARDS PROLIFERATE: (SEE: A/C CHARGERS, CHARACTER ENCODINGS, INSTANT MESSAGING, ETC.) Panel 1: SITUATION: THERE ARE I4 COMPETING STANDARDS. Panel 2: character 1: 14?! RIDICULOUS! WE NEED TO DEVELOP ONE UNIVERSAL STANDARD THAT COVERS EVERYONE'S USE CASES. character 2: YEAH! Panel 3: SOON: SITUATION: THERE ARE I5 COMPETING STANDARDS. YEAH!
A plot showing water access by development level from 2000 to 2022. High-income countries have nearly 100% safe access, while access steadily worsens for lower income levels, but steadily improves over time.
A face in a black mask with rainbow gem eyeshadow, standing in front of a giant trans flag
METRICS DOMINATE EXISTING ASSESSMENT PRACTICES, BUT MOST RESEARCHERS DESIRE BALANCE • The majority of respondents told us they are evaluated either entirely or mostly based on metrics — primarily publication metrics. • All researchers want assessment to be more representative of their contributions other than research outputs. • Compared to existing practices, there is also a clear desire to reduce reliance on metrics. Plot showing that people think current assessment is skewed towards quantitative measures, but want more balance At present, what is the mix of quantitative and qualitative methods for how your work is evaluated? (n=4,910) In an ideal research assessment process, what would you suggest should be the weighting? (n=4,869)
Formative Appraisal Where things have gone well - Major establishment bodies bought in - Withstood public opposition - Grown the coalition - (Positive) unanticipated effects and 'small wins' Challenges - Expecting too much, too soon (e.g. over half of sector claimed no meaningful benefits yet. Source: Culture Barometer 2024) - Lack of awareness (53% of PhD researchers not aware of program) - Persuading 'rank and file' gatekeepers and wider academic community - Engaging with research fields and publishing reform movement
DRIVERS OF ASSESSMENT REFORM - Open research - Research Integrity - Equity, Diversion & Inclusion - Research Culture - Research careers
Which elements of open science are part of the documented strategic approach followed by your organisation? Open access to research articles Findable, Accessible, Interoperable, Re-usable (FAIR) | research data (including for example data management plans) Open access to research data (including for example metadata) Open access to research books Open research infrastructure Stakeholder engagement with industry partners, policy makers, and/or the public (including for example science communication) Open source research software, code, and tools (including for example software management plans) Citizen science Open evaluation (including for example open peer review. open and transparent indicators/metrics) Open research methods (such as pre-registration of study designs, open protocols and workflows) Service and leadership (such as engagement in open science policy development, teaching, and/or supervision) Open hardware Open education Outcome from a 2024 survey among Science Europe Member Organisations (Europe's foremost public research funding and research performing organisations) on their strategic approaches to, and research assessment of, Open Science. See scieur.org/ra-os-survey for more information.
The African open science landscape, a slide with logos of organisations like Africa Open Science Hardware and AfricArXiv
Plots showing significant but still quite small increases from 2015 to 2022 in prevalence of preregistration, open materials, data, analysis scripts, open access, conflict of interest statements, and funding statements
Results Change Every Run (Unstable) 13.4% Manual Steps Not Documented 15.7% Starts from Intermediate (Undocumented) Data 15.7% Coding Mistakes 19.8% Paper Exhibits are Outdated 35.5%
A plot that shows changes from first submission to subsequent submissions Improving Science in the World Bank Authors Learn by Doing Packages Ran at First Try: 42.5 to 55.4% Packages with Manual Steps Not Documented: 20.9 to 8.8% Package Starts From Intermediate (undocumented) data: 18.6 to 5.9%
Reproducing Al studies - results Data only 33% success 67% failure Code and data 86% success 14% failure Code is important - not its documentation Data documentation is important
A panel with John Ioannides speaking in fromt of a slide that reads: "Is science currently overall more beneficial or harmful to humans?" In 2005 "Undoubtedly yes!" In 2025 "I hope so, but I am not so sure" See conflicts of interest (?), SARS-CoV-2 from a lab (?), lockdowns (?), loss of trust, challenge to Mertonian values (disinterestedness, communality, universalism, organized skepticism)
A person standing on their head with no hands in front of the famous Rotterdam statue of Santa holding a bell and a Christmas tree (get your mind out of the gutter…)
In both sessions, blood samples (20 mL) were first collected in which we assessed levels of FSH, LH, total and free testosterone, SHBG, estradiol, and progesterone. Participants thereafter filled out online surveys on sexual function, general well-being and depressive symptoms. These results are reported in Zethraeus et al. (2016) and Zethraeus et al. (2017). After these surveys, participants were given the online survey on facial preferences. In the follow-up session, participants also participated in an economic experiment after the survey on facial preferences. The results from that experiment are reported in Ranehill et al. (2018). In this experiment, we also measured 2D:4D and tested if 2D:4D was associated with economic preferences; those results are reported in Parslow et al. (2019).
A person doing a handstand in front of a church door. The photo is taken from below so the church looms in perspective.
Plot showing years since publication on the x-axis (0-25) and probability of data recovery on the y-axis (0-1). The curve starts around 0.8 prob at 0 years and rapidly declines to 0 by 20 years.
Dear [MP of my local authority], I am writing to express my serious concern about the UK Supreme Court’s recent judgment in For Women Scotland Ltd v The Scottish Ministers [2025] UKSC 16. In allowing the appeal, the Court concluded that the term “woman" refers exclusively to biological females, excluding trans women with a Gender Recognition Certificate. This interpretation not only undermines the rights and dignity of transgender people, but also sets a dangerous precedent for how legal protections for all minority groups may be interpreted in future. The judgment effectively denies legal recognition of acquired gender in a context where such recognition is fundamental to the safety, inclusion, and equality of trans people. By reducing the legal meaning of “woman” to biology alone, the Court has reinforced a rigid and exclusionary framework that threatens to marginalise trans individuals across public life including employment, healthcare, and political participation. This decision risks emboldening discrimination under the guise of clarity. We are already seeing an immediate rise in the discriminationory group activity which is well publicised on national media. More broadly, this decision signals a retreat from inclusive statutory interpretation and may pave the way for similar restrictions to be placed on other protected characteristics, such as religion, sexual orientation, and disability. If fundamental rights can be narrowed in this way, then the principles of dignity, autonomy, and fairness which underpin the equality law are at risk. I urge you to raise this matter in Parliament and call for legislative clarification to ensure that equality law remains inclusive, humane, and fit for modern society. Regards, Name Full address
A photo of a hard copy of an xkcd comic (https://xkcd.com/303): a line drawing of two people sword fighting on rolling chairs. Speech bubbles out of a door say: “HEY! GET BACK TO WORK!” “COMPILING!” “OH. CARRY ON.” The text above the comic says: THE #1 PROGRAMMER EXCUSE FOR LEGITIMATELY SLACKING OFF: "MY CODE'S COMPILING."
The distribution of imprecise p-values in results sections. Values with < are shown in red, and > are shown in blue. Notably there are many instances of `p < .01`, `p < .05`, and `p > .25`, with smaller peaks at `p > .05`, `p < .02` and `p < .005`.
A chart showing the relationship between citation count (plotted on the x-axis) and field-weighted citation impact (FWCI; plotted on the y-axis). The relation is strongly positive and linear, with some variation. The papers are represented by points with the colour by year of publication (2014-2024), showing that the papers above the regression line tend to be more recent, and those below the line tend to be older.
A chart showing the fraction of published papers received each year by Psychological Science, from 2012 to 2023, that have open data, open materials, or preregistration badges. The number is mostly increasing each year, with a sharp decrease in 2022.
A person sitting on a chair with one leg tucked underneath and the other foot on a table
Participants needed For a qualitative study on seeking emotional support from Generative AI Chatbots Eligibility criteria ⁃ Adult (18+) ⁃ Currently using GenAI chatbots for emotional support ⁃ Comfortable discussing emotional experiences Commitment and benefits ⁃ 3h of participation over 2 weeks ⁃ £30-39 voucher compensation Social AI: social-cdt.org University of Glasgow
Water splashing over the v-shaped rusty edge of a canal lock into the canal far below, photo taken from above.
Time to go back home Research question: There is a strike, is the train coming? Probability grid
META-RESEARCH Questionable research practices may have little effect on replicability This article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indicate that the base rate of true effects is the major factor that determines the replication rate of scientific results. Specifically, for purely statistical reasons, replicability is low in research domains where true effects are rare (e.g., search for effective drugs in pharmacology). This point is under-appreciated in current scientific and media discussions of replicability, which often attribute poor replicability mainly to QRPs.
Five Possible CFA Models Based on Individual-Level Measures Individual-level construct Model 1: aggregate Model 2: within cluster Cluster-level construct Model 3: Between cluster (shared) Model 4: Within and between cluster (configural) Model 5: simultaneous shared and configural
Scrutinise previous work measures -> Interpret existing findings & Increase reuse -> Let’s build cumulative evidence in psychology
A banner in Chinese, with the English translation “Translate server error”
If we only reward scientists for what they do as individuals, they will compete with each other, instead of helping each other to do better work together. Tiokhin, Panchanathan, Smaldino, Lakens, 2023
A timelapse drawing of a scene from Kiki’s Delivery Service, with Tombo and Kiki on a flying bike
A banner in Chinese, with the English translation “Translate server error”
Aurelia Munene speaking with a slide of organisations involved in Open Peer Review in Africa
A key made to look like electronic compionents, against a background of 0s and 1s
A 4-section grid with the left-right axis "Most useful when studying" to "Most useful when working" and the up-down axis "practical steps" to "theoretical knowledge". Tutorials are in the upper left, How-to guides in the upper right, Explanation in the lower left, and Reference in the lower right.
Three hexagonal stickers with red backgrounds reading “rethinking”. The top has a ~ and the bottom two have a picture of a golem.
A power curve showing the probability of detecting effects between 0.1 and 0.5 increases as raters increase up to 500
Simulated data showing the distribution of effect sizes for the difference between pairs of composites sampled from the same distribution (i.e., no real effect). Points show the median unsigned effect size. As the number of stimuli per composite decreases, the variability increases.
Average female and male faces made using morphing.