False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant

Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn

Research output: Indexed journal article Articlepeer-review

4481 Citations (Scopus)

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤.05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

Original languageEnglish
Pages (from-to)1359-1366
Number of pages8
JournalPsychological Science
Volume22
Issue number11
DOIs
Publication statusPublished - Nov 2011
Externally publishedYes

Keywords

  • disclosure
  • methodology
  • motivated reasoning
  • publication

Fingerprint

Dive into the research topics of 'False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant'. Together they form a unique fingerprint.

Cite this