Estimating the Reproducibility of Psychological Science.

Open Science Collaboration, Manuela Thomae, Michael Wood

Research output: Contribution to journalArticleResearchpeer-review

4 Downloads (Pure)

Abstract

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Original languageEnglish
JournalScience
Volume349
Issue number6251
DOIs
Publication statusPublished - 28 Aug 2015

Keywords

  • empirical analysis
  • error analysis
  • innovation
  • meta-analysis
  • psychology
  • research method
  • research work
  • Article
  • confidence interval
  • correlational study
  • effect size
  • prediction
  • priority journal
  • publishing
  • reproducibility
  • sampling
  • selection bias
  • social psychology
  • statistical analysis
  • statistical significance

Cite this

Open Science Collaboration, Thomae, M., & Wood, M. (2015). Estimating the Reproducibility of Psychological Science. Science, 349(6251). https://doi.org/10.1126/science.aac4716
Open Science Collaboration ; Thomae, Manuela ; Wood, Michael. / Estimating the Reproducibility of Psychological Science. In: Science. 2015 ; Vol. 349, No. 6251.
@article{0458891f323b49f2a55ff1747a237d87,
title = "Estimating the Reproducibility of Psychological Science.",
abstract = "Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47{\%} of original effect sizes were in the 95{\%} confidence interval of the replication effect size; 39{\%} of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68{\%} with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.",
keywords = "empirical analysis, error analysis, innovation, meta-analysis, psychology, research method, research work, Article, confidence interval, correlational study, effect size, prediction, priority journal, publishing, reproducibility, sampling, selection bias, social psychology, statistical analysis, statistical significance",
author = "{Open Science Collaboration} and Manuela Thomae and Michael Wood",
year = "2015",
month = "8",
day = "28",
doi = "10.1126/science.aac4716",
language = "English",
volume = "349",
number = "6251",

}

Open Science Collaboration, Thomae, M & Wood, M 2015, 'Estimating the Reproducibility of Psychological Science.', Science, vol. 349, no. 6251. https://doi.org/10.1126/science.aac4716

Estimating the Reproducibility of Psychological Science. / Open Science Collaboration; Thomae, Manuela; Wood, Michael.

In: Science, Vol. 349, No. 6251, 28.08.2015.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Estimating the Reproducibility of Psychological Science.

AU - Open Science Collaboration, null

AU - Thomae, Manuela

AU - Wood, Michael

PY - 2015/8/28

Y1 - 2015/8/28

N2 - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

AB - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

KW - empirical analysis

KW - error analysis

KW - innovation

KW - meta-analysis

KW - psychology

KW - research method

KW - research work

KW - Article

KW - confidence interval

KW - correlational study

KW - effect size

KW - prediction

KW - priority journal

KW - publishing

KW - reproducibility

KW - sampling

KW - selection bias

KW - social psychology

KW - statistical analysis

KW - statistical significance

U2 - 10.1126/science.aac4716

DO - 10.1126/science.aac4716

M3 - Article

VL - 349

IS - 6251

ER -

Open Science Collaboration, Thomae M, Wood M. Estimating the Reproducibility of Psychological Science. Science. 2015 Aug 28;349(6251). https://doi.org/10.1126/science.aac4716