Empirical Questions in Metascience: A List for Funders and Researchers
A Policy-Relevant Research Agenda
I first became interested in metascience in 2019, while contributing to a “systematic review” - a process of identifying, critically evaluating and synthesising research findings on a specific topic. Over the next few years, more academic research experience, spanning in vitro and in vivo wet lab research, bioinformatics and clinical medical research, made metascience feel even more important.
When I found myself working on science policy, in politics and later at a think tank, I tried to bring ideas from academic metascience to policy. But by and large, for policy decisions by governments and arms-length funders, I felt that most of the evidence available was sparse, narrow or of low rigour.
Over the past couple of years I’ve kept a rough list of empirical metascience questions I'd be particularly excited to find answers to (or in some cases, better answers to).
Attending Metascience 2025 last week inspired me to clean up the list and make the questions public.
I am not fortunate enough to be paid to conduct or fund academic metascience research - but if you are, you might find this helpful.
Some questions might not be relevant for some disciplines, and many answers might vary by discipline. For some of these questions, I have suggested study designs that might be suitable.
I’m most interested in what these answers look like for economics, biology and biomedical science, for a mix of three reasons - these are the disciplines I engage with the most, these disciplines have a lot of room to improve, and I think these disciplines are particularly important for national and global flourishing.
I will maintain this as a live document with regular updates.
Funding
How effective a proxy for risk aversion would “proportion of published null results from funded work” be, when comparing funding organisations and individual grantmakers?
To what extent do the same researchers consistently do the most disruptive research?
To what extent is “disruptive” research contingent on “incremental” research, and how clearly can they be disentangled? (citation analysis)
Project Selection, Risk Aversion and Disruptiveness
What proportion of academics would change the focus of their research project if they felt that journals would not penalise null results? (survey)
What proportion of academics would change the focus of their research project if they felt that funders would not penalise null results? (survey)
Do academics become more likely to publish null results after they receive tenure or after they become full professors? (pre-post study)
How strong is outcome bias amongst peer reviewers, when presented with the same fictional study but with statistically significant vs null results? (randomised trial)
Are research findings being spread out over more papers than before because of publication pressures, and does this help explain a possible decline in perceived disruptiveness?
Open Science Practices
What are the effects of preregistration on the prevalence of questionable research practices and on reproducibility? (randomised trial)
What are the effects of preprinting on the prevalence of questionable research practices and on reproducibility? (randomised trial)
What are the effects of reporting guidelines on the prevalence of questionable research practices and on reproducibility? (randomised trial)
What training interventions work to reduce the prevalence of questionable research practices and improve reproducibility? (randomised trials)
How does the prevalence of questionable research practices compare in papers published by the private sector vs by academics? (comparing the effectiveness of profit and academic prestige at encouraging rigour)
Replication, Reproduction and Retraction
How does a high profile replication failure affect the use of open science practices by “exposed” researchers? (difference-in-differences)
“Exposed” researchers could be considered to be those at the same department or institution, or in the same academic subfield, as a team which authored a paper that could not be replicated.
“Open science practices” could include preregistration, public sharing of data, and use of reporting guidelines.
How does a high profile retraction affect the prevalence of questionable research practices in future work by “exposed” researchers? (difference-in-differences).
The File Drawer Problem
How many discrete research projects has the average academic conducted but not published the results of, due to null results? (survey)
Peer Review
How reliable are LLMs at identifying questionable research practices?
Collaboration
What are the effects of different conference formats on a) the number and b) the disruptiveness of future collaborations by attendees? (randomised trials)
What is the effect of active, online academic communities on a) the number and b) the disruptiveness of research collaborations? (can we identify any effects of the academic exodus from Twitter / X?) (regression discontinuity)
How many research ideas on average do academics keep secret or private at any given time, due to fear of “scooping”? (survey)
What shared or common features, if any, have existed in teams and labs where research led to Nobel Prizes? (qualitative interviews)
How does the performance of university researchers (who deliver teaching alongside research) compare to researchers at independent institutes (with more limited teaching functions, such as Britain’s “Public Sector Research Establishments”)? (this could empirically test the extent of synergy between teaching and frontier research)
Other AI
How effective are LLMs at reformatting research manuscripts to the requirements of different academic journals?
Publish or Perish
Is there a correlation at the researcher level, departmental level or institution level between frequency of publication and the prevalence of questionable research practices? (LLM-assisted analysis)
How does the publication frequency of award-winning researchers compare to that of the median researcher in their academic subfield, department and institution?
Management
How prevalent are “structured management practices” in academic labs and teams? (survey)
Economists John Van Reenen, Nicholas Bloom and Raffaella Sadun have shown that on average, structured management practices (like KPIs and performance targets) improve the productivity of organisations.
Senior researchers end up spending lots of their time acting as “reluctant managers” for junior researchers, rather than being able to do their own research. Presumably, “reluctant” managers are not very good managers, and management training for academics seems uncommon.
What proportion of academic teams use the single “principle investigator” (PI) model, compared to non-traditional management models (like co-PIs and rotating PIs)? (cross-sectional descriptive study using either web-scraped data or a survey)
How does the average performance of academic teams using non-traditional management models (like co-PIs or rotating-PIs) compare to that of matched teams using the single-PI model? (matched observational cohort study, focusing on publications, citations and / or patenting as outcomes)
Update 15/7/25: added “Publish or Perish” section.
Update 1/8/25: added “Management” section.
Click here for a list of AI Products we need to build for metascience.
I use this Substack for rough thinking around policy and politics, rather than finalised policy proposals. If you disagree with ideas in this post, think I’ve missed something obvious or have any other feedback, I’d love to hear from you!