Abstract

A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, ‘double dipping’, the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we can apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. I will explain how circularity can be safely avoided in neuroimaging experiments. In the last 15 minutes, I will broaden the scope beyond neuroscience and discuss how selection at the level of studies and publications can introduce spurious results into the literature and create biases affecting entire fields of science.