Abstract

Within neuroscience, psychology and neuroimaging, it is typical to run an experiment on a sample of participants and then apply statistical tools to quantify and infer an effect of the experimental manipulation in the population from which the sample was drawn. Whereas the current focus is on average effects (i.e. the population mean, assuming a normal distribution¹), it is equally valid to ask the alternative question of how typical is the effect in the population²? That is, we infer an effect in each individual participant in the sample, and from that infer the prevalence of the effect in the population[3–6]. We propose a novel Bayesian method to estimate such population prevalence, based on within-participant null-hypothesis significance testing (NHST). Applying Bayesian population prevalence estimation in studies sufficiently powered for NHST within individual participants could address many of the issues recently raised regarding replicability⁷. Bayesian prevalence provides a population level inference currently missing for designs with small numbers of participants, such as traditional psychophysics or animal electrophysiology[8,9]. Since Bayesian prevalence delivers a quantitative estimate with associated uncertainty, it avoids reducing an entire experiment to a binary inference on a population mean¹⁰.

1. Holmes, A. & Friston, K. Generalisability, random effects and population inference. Neuroimage 7, (1998).

2. Friston, K. J., Holmes, A. P. & Worsley, K. J. How Many Subjects Constitute a Study? NeuroImage 10, 1–5 (1999).

3. Friston, K. J., Holmes, A. P., Price, C. J., Büchel, C. & Worsley, K. J. Multisubject fMRI Studies and Conjunction Analyses. NeuroImage 10, 385–396 (1999).

4. Rosenblatt, J. D., Vink, M. & Benjamini, Y. Revisiting multi-subject random effects in fMRI: Advocating prevalence estimation. NeuroImage 84, 113–121 (2014).

5. Allefeld, C., Görgen, K. & Haynes, J.-D. Valid population inference for information-based imaging: From the second-level t-test to prevalence inference. NeuroImage 141, 378–392 (2016).

6. Donhauser, P. W., Florin, E. & Baillet, S. Imaging of neural oscillations with embedded inferential and group prevalence statistics. PLOS Computational Biology 14, e1005990 (2018).

7. Benjamin, D. J. et al. Redefine statistical significance. Nature Human Behaviour 2, 6 (2018).

8. Neuroscience, S. for. Consideration of Sample Size in Neuroscience Studies. J. Neurosci. 40, 4076–4077 (2020).

9. Smith, P. L. & Little, D. R. Small is beautiful: In defense of the small-N design. Psychon Bull Rev 25, 2083–2101 (2018).

10. McShane, B. B., Gal, D., Gelman, A., Robert, C. & Tackett, J. L. Abandon Statistical Significance. The American Statistician 73, 235–245 (2019).