TY - JOUR
T1 - False-positive psychology
T2 - Undisclosed flexibility in data collection and analysis allows presenting anything as significant
AU - Simmons, Joseph P.
AU - Nelson, Leif D.
AU - Simonsohn, U.
PY - 2011/11
Y1 - 2011/11
N2 - In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤.05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
AB - In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤.05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
KW - disclosure
KW - methodology
KW - motivated reasoning
KW - publication
UR - http://www.scopus.com/inward/record.url?scp=80555145867&partnerID=8YFLogxK
U2 - 10.1177/0956797611417632
DO - 10.1177/0956797611417632
M3 - Article
AN - SCOPUS:80555145867
SN - 0956-7976
VL - 22
SP - 1359
EP - 1366
JO - Psychological Science
JF - Psychological Science
IS - 11
ER -