Stimulus Sampling Reimagined: Designing Experiments with Mix-and-Match, Analyzing Results with Stimulus Plots

Stimuli selection in psychology experiments is typically unsystematic, undocumented, and irreproducible. This makes confounds likely to arise. The statistical analysis of psychology experiments with multiple stimuli, in turn, is typically reported at the aggregate level, averaging across stimuli. This makes confounds unlikely to be detected. Here, we propose changing both the design and analysis of psychology experiments. We introduce “Mix-and-Match,” a procedure to systematically and reproducibly stratify-sample stimuli, and “Stimulus Plots,” a visualization to report stimulus-level results, contrasting observed with expected variation. We apply both innovations to published studies demonstrating how things would be different with our reimagined approach to stimulus sampling. Finally, we introduce a Mix-and-Match Disclosure Form we propose authors rely on to communicate the design of their studies. (PsycInfo Database Record (c) 2025 APA, all rights reserved)