99% Impossible: A Valid, or Falsifiable, Internal Meta-Analysis

Joachim Vosgerau, Uri Simonsohn, Leif D. Nelson, Joseph P. Simmons

Producció científica: Article en revista indexadaArticleAvaluat per experts

54 Cites (Scopus)

Resum

Several researchers have relied on, or advocated for, internal meta-analysis, which involves statisticallyaggregating multiple studies in a paper to assess their overall evidential value. Advocates of internalmeta-analysis argue that it provides an efficient approach to increasing stat istical power and solving thefile-drawer problem. Here we show that the validity of internal meta-analysis rests on the assumption thatno studies or analyses were selectively reported. That is, the technique is only valid if (a) all conductedstudies were included (i.e., an empty file drawer), and (b) for each included study, exactly one analysiswas attempted (i.e., there was no p-hacking). We show that even very small doses of selective reportinginvalidate internal meta-analysis. For example, the kind of minimal p-hacking that increases thefalse-positive rate of 1 study to just 8% increases the false-positive rate of a 10-study internalmeta-analysis to 83%. If selective reporting is approximately zero, but not exactly zero, then internalmeta-analysis is invalid.

Idioma originalAnglès
Pàgines (de-a)1628-1639
Nombre de pàgines12
RevistaJournal of Experimental Psychology: General
Volum148
Número9
DOIs
Estat de la publicacióPublicada - 2019

Fingerprint

Navegar pels temes de recerca de '99% Impossible: A Valid, or Falsifiable, Internal Meta-Analysis'. Junts formen un fingerprint únic.

Com citar-ho