Replication is a foundational principle for the credibility of science [Romero, 2019]. However, psychology and neuroscience are currently experiencing a replicability and reproducibility crisis, evoked by the realization that studies failed to replicate or reproduce some previously popularized findings [Klein et al., 2014; Malich & Rehmann-Sutter, 2022], such as the Marshmallow effect [De Posada & Singer, 2005], the Mozart effect [Campbell, 2000], “feeling the future” using extrasensory perception [Bem, 2011; Rabeyron, 2020; Ritchie, Wiseman, & French, 2012], trait construct priming [Bargh, Chen, & Burrows,1996; Doyen, Klein, Pichon, & Cleeremans, 2012; Klein et al., 2014; Stroebe, 2019], or the power pose effect [Carney, Cuddy, & Yap, 2010].

Neuroscientific findings have been shown to hold a certain allure for the general public, such that non-experts trust statements more when allegedly backed by neuroscience [Weisberg, Keil, Goodstein, Rawson, & Gray, 2008]. Scientists thus have the responsibility to exert caution when communicating single findings to avoid sensationalism and to perform rigorous research despite external pressure to produce results.

Considering sample size is important for generalizable and sound inferences

Neuroscientists and psychologists fundamentally rely on statistical theories and methods [Chen, 2019], whether to map brain function to structure, study the connectivity between brain regions, model brain dynamics, or detect aberrances. A single finding might be observed due to chance, meaning that sometimes we might observe an unusual result in a sample that does not accurately reflect the behavior of the population. The outcome of a single experiment can stray quite far from the expected or theoretical average, especially when dealing with a small sample size or a situation with high variability.

However, according to Bernoulli’s law of large numbers, when we gather data from a large sample of individuals, the whole population’s average for a certain trait can be estimated more accurately [Blume & Royall, 2003; Bolthausen & Wüthrich, 2013]. In simpler terms, the larger the sample size, the better the average of the population’s characteristics can be represented.

Therefore, it is advisable to record many observations, either on the individual or using a large sample size, to draw generalizable conclusions for the individual or a population, respectively. However, in human studies, lack of time, funding, and human resources often restrict sample sizes to humble numbers, which limits statistical power and can lead to spurious and non-replicable results [Button et al., 2013; Nee, 2019; Szucs & Ioannidis, 2020].

Current scientific policies pressure researchers to publish

The pressure to publish leads to a focus on novelty and a bias towards the publication of positive findings, which in turn can incentivize bad practices that increase the likelihood of false positives, such as HARKing, cherry-picking, p-hacking, fishing and data-dredging, or even fraud [Andrade, 2021; Yong, 2012; Stroebe, 2019]. Moreover, researchers may not adhere to the most up-to-date recommendations and good analysis practices [Nieuwenhuis, Forstmann, & Wagenmakers, 2013; Vul, Harris, Winkielman, & Pashler, 2009].

However, it should be noted that not only poor science is at fault – intrinsically, some hypotheses will be false when exploring the unknown and our current statistical methods will per definition lead to some false positives (“the base rate fallacy”) [Bernard, 2023; Bird, 2021; Hunter, 2017]. Either way, it is important to establish the necessary methodology and environment to allow for high-quality research. Let us all contribute to a scientific community that fosters data sharing, knowledge exchange, collaboration, and mutual support.

Big data and open science initiatives aim to mitigate challenges

The (Neuro)science community is now evolving to try to meet and mitigate these challenges with big data and open science initiatives. With these approaches, we can view the spurred discussion and endeavors as a chance at and leverage for better science, instead of just a crisis [Munafo, 2022]. To promote this movement, in this Flux Blog, we provide an annotated collection of literature, guidelines, and databases in reference to this timely topic.

If you have questions, comments, or would like to add to this collection, please contact us, we would appreciate your participation! 

Please share your feedback here and/or leave us a note in the comment section!

References:

Andrade, C. (2021). HARKing, cherry-picking, p-hacking, fishing expeditions, and data dredging and mining as questionable research practices. The Journal of Clinical Psychiatry, 82(1), 25941. doi: https://doi.org/10.4088/JCP.20f13804 

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of personality and social psychology, 71(2), 230. doi: https://doi.org/10.1037/0022-3514.71.2.230

Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407. doi: https://doi.org/10.1037/a0021524 

Bernard, C. (2023). Stop Reproducing the Reproducibility Crisis. Eneuro, 10(2).doi: https://doi.org/10.1523/ENEURO.0032-23.2023 

Bird, A. (2021). Understanding the replication crisis as a base rate fallacy. The British Journal for the Philosophy of Science. doi: https://doi.org/10.1093/bjps/axy051 

Blume, J. D., & Royall, R. M. (2003). Illustrating the law of large numbers (and confidence intervals). The American Statistician, 57(1), 51-57. doi: https://doi.org/10.1198/0003130031081 

Bolthausen, E., & Wüthrich, M. V. (2013). Bernoulli's law of large numbers. ASTIN Bulletin: The Journal of the IAA, 43(2), 73-79. doi: https://doi.org/10.1017/asb.2013.11 

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience, 14(5), 365-376. doi: https://doi.org/10.1038/nrn3475 

Carney, D. R., Cuddy, A. J., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological science, 21(10), 1363-1368. doi: https://doi.org/10.1177/0956797610383437 

Chén, O. Y. (2019). The roles of statistics in human neuroscience. Brain sciences, 9(8), 194. doi: https://doi.org/10.3390/brainsci9080194

Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it's all in the mind, but whose mind?. PloS one, 7(1), e29081. doi: https://doi.org/10.1371/journal.pone.0029081

Hunter, P. (2017). The reproducibility “crisis” Reaction to replication crisis should not stifle innovation. EMBO reports, 18(9), 1493-1496. doi: https://doi.org/10.15252/embr.201744876   

Klein, R. A., Ratliff, K. A., Vianello, M., Adams Jr, R. B., Bahník, Š., Bernstein, M. J., ... & Nosek, B. A. (2014). Investigating variation in replicability. Social psychology. doi: https://doi.org/10.1027/1864-9335/a000178

National Academies of Sciences, Engineering, and Medicine (2019). Understanding Reproducibility and Replicability. Reproducibility and Replicability in Science. https://www.ncbi.nlm.nih.gov/books/NBK547546/ 

Nieuwenhuis, S., Forstmann, B. & Wagenmakers, EJ (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nat Neurosci 14, 1105–1107. https://doi.org/10.1038/nn.2886 

Nee, D. E. (2019). fMRI replicability depends upon sufficient individual-level data. Communications biology, 2(1), 130. doi: https://doi.org/10.1038/s42003-018-0073-z 

Malich, L., & Rehmann-Sutter, C. (2022). Metascience is not enough–a plea for psychological humanities in the wake of the replication crisis. Review of General Psychology, 26(2), 261-273. doi: https://doi.org/10.1177/10892680221083876

Munafò, M. R., Chambers, C., Collins, A., Fortunato, L., & Macleod, M. (2022). The reproducibility debate is an opportunity, not a crisis. BMC Research Notes, 15(1), 1-3. doi: https://doi.org/10.1186/s13104-022-05942-3 

Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American journal of epidemiology, 163(9), 783-789. doi: https://doi.org/10.1093/aje/kwj093 

Rabeyron, T. (2020). Why most research findings about psi are false: The replicability crisis, the psi paradox and the myth of Sisyphus. Frontiers in Psychology, 11, 562992. doi: https://doi.org/10.3389/fpsyg.2020.562992 

Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Replication, replication, replication. The Psychologist.

Romero, F. (2019). Philosophy of science and the replicability crisis. Philosophy Compass, 14(11), e12633. doi: https://doi.org/10.1111/phc3.12633 

Stroebe, Wolfgang. "What can we learn from many labs replications?." Basic and Applied Social Psychology 41, no. 2 (2019): 91-103. doi: https://doi.org/10.1080/01973533.2019.1577736 

Szucs, D., & Ioannidis, J. P. (2020). Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals. NeuroImage, 221, 117164. doi: https://doi.org/10.1016/j.neuroimage.2020.117164 

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science, 4(3), 274–290. https://doi.org/10.1111/j.1745-6924.2009.01125.x

Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The seductive allure of neuroscience explanations. Journal of cognitive neuroscience, 20(3), 470-477. doi: https://doi.org/10.1162%2Fjocn.2008.20040 

Yong, E. (2012). Bad copy. Nature, 485(7398), 298. doi: https://doi.org/10.1038/485298a