99% Impossible: A Valid, or Falsifiable, Internal Meta-Analysis

Several researchers have relied on, or advocated for, internal meta-analysis, which involves statisticallyaggregating multiple studies in a paper to assess their overall evidential value. Advocates of internalmeta-analysis argue that it provides an efficient approach to increasing stat istical power and solving thefile-drawer problem. Here we show that the validity of internal meta-analysis rests on the assumption thatno studies or analyses were selectively reported. That is, the technique is only valid if (a) all conductedstudies were included (i.e., an empty file drawer), and (b) for each included study, exactly one analysiswas attempted (i.e., there was no p-hacking). We show that even very small doses of selective reportinginvalidate internal meta-analysis. For example, the kind of minimal p-hacking that increases thefalse-positive rate of 1 study to just 8% increases the false-positive rate of a 10-study internalmeta-analysis to 83%. If selective reporting is approximately zero, but not exactly zero, then internalmeta-analysis is invalid.