Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Comparing Analysis Blinding With Preregistration in the Many-Analysts Religion Project

Comparing Analysis Blinding With Preregistration in the Many-Analysts Religion Project In psychology, preregistration is the most widely used method to ensure the confirmatory status of analyses. However, the method has disadvantages: Not only is it perceived as effortful and time-consuming, but reasonable deviations from the analysis plan demote the status of the study to exploratory. An alternative to preregistration is analysis blinding, in which researchers develop their analysis on an altered version of the data. In this experimental study, we compare the reported efficiency and convenience of the two methods in the context of the Many-Analysts Religion Project. In this project, 120 teams answered the same research questions on the same data set, either preregistering their analysis (n = 61) or using analysis blinding (n = 59). Our results provide strong evidence (Bayes factor [BF] = 71.40) for the hypothesis that analysis blinding leads to fewer deviations from the analysis plan, and if teams deviated, they did so on fewer aspects. Contrary to our hypothesis, we found strong evidence (BF = 13.19) that both methods required approximately the same amount of time. Finally, we found no and moderate evidence on whether analysis blinding was perceived as less effortful and frustrating, respectively. We conclude that analysis blinding does not mean less work, but researchers can still benefit from the method because they can plan more appropriate analyses from which they deviate less frequently. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Advances in Methods and Practices in Psychological Science SAGE

Comparing Analysis Blinding With Preregistration in the Many-Analysts Religion Project

Download PDF
19 pages

Loading next page...
 
/lp/sage/comparing-analysis-blinding-with-preregistration-in-the-many-analysts-Es3AG5u20Q
Publisher
SAGE
Copyright
© The Author(s) 2023
ISSN
2515-2459
eISSN
2515-2467
DOI
10.1177/25152459221128319
Publisher site
See Article on Publisher Site

Abstract

In psychology, preregistration is the most widely used method to ensure the confirmatory status of analyses. However, the method has disadvantages: Not only is it perceived as effortful and time-consuming, but reasonable deviations from the analysis plan demote the status of the study to exploratory. An alternative to preregistration is analysis blinding, in which researchers develop their analysis on an altered version of the data. In this experimental study, we compare the reported efficiency and convenience of the two methods in the context of the Many-Analysts Religion Project. In this project, 120 teams answered the same research questions on the same data set, either preregistering their analysis (n = 61) or using analysis blinding (n = 59). Our results provide strong evidence (Bayes factor [BF] = 71.40) for the hypothesis that analysis blinding leads to fewer deviations from the analysis plan, and if teams deviated, they did so on fewer aspects. Contrary to our hypothesis, we found strong evidence (BF = 13.19) that both methods required approximately the same amount of time. Finally, we found no and moderate evidence on whether analysis blinding was perceived as less effortful and frustrating, respectively. We conclude that analysis blinding does not mean less work, but researchers can still benefit from the method because they can plan more appropriate analyses from which they deviate less frequently.

Journal

Advances in Methods and Practices in Psychological Science – SAGE

Published: Jan 1, 2023

Keywords: Open Science; metascience; replication crisis; many analysts; Open Data; Open Materials; Preregistration

References

  • Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies
    Claesen, A.; Gomes, S.; Tuerlinckx, F.; Vanpaemel, W.
  • Flexible yet fair: Blinding analyses in experimental psychology
    Dutilh, G.; Sarafoglou, A.; Wagenmakers, E.-J.
  • A test of the diffusion model explanation for the worst performance rule using preregistration and blinding
    Dutilh, G.; Vandekerckhove, J.; Ly, A.; Matzke, D.; Pedroni, A.; Frey, R.; Rieskamp, J.; Wagenmakers, E.-J.
  • Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling
    Gelfand, A. E.; Smith, A. F.; Lee, T.-M.
  • A tutorial on bridge sampling
    Gronau, Q. F.; Sarafoglou, A.; Matzke, D.; Ly, A.; Boehm, U.; Marsman, M.; Leslie, D. S.; Forster, J. J.; Wagenmakers, E.-J.; Steingroever, H.
  • Bridgesampling: An R package for estimating normalizing constants
    Gronau, Q. F.; Singmann, H.; Wagenmakers, E.-J.
  • Nasa-Task Load Index (NASA-TLX); 20 years later
    Hart, S. G.
  • Documenting contributions to scholarly articles using CRediT and
    Holcombe, A. O.; Kovacs, M.; Aust, F.; Aczel, B.
  • A many-analysts approach to the relation between religiosity and well-being
    Hoogeveen, S.; Sarafoglou, A.; Aczel, B.; Aditya, Y.; Alayan, A.; Allen, P.; Altay, S.; Alzahawi, S.; Amir, Y.; Anthony, F.-V.; Appiah, O.; Atkinson, Q. D.; Baimel, A.; Balkaya-Ince, M.; Balsamo, M.; Banker, S.; Bartoš, F.; Becerra, M.; Beffara, B.; . . . Wagenmakers, E.-J.
  • Advancing the cognitive science of religion through replication and open science
    Hoogeveen, S.; van Elk, M.
  • Bayesian model selection using encompassing priors
    Klugkist, I.; Kato, B.; Hoijtink, H.
  • Zero-inflated Poisson regression, with an application to defects in manufacturing
    Lambert, D.
  • Hide results to seek the truth: More fields should, like particle physics, adopt blind analysis to thwart bias
    MacCoun, R.; Perlmutter, S.
  • A manifesto for reproducible science
    Munafò, M.; Nosek, B. A.; Bishop, D.; Button, K.; Chambers, C.; Du Sert, N.; Simonsohn, U.; Wagenmakers, E.-J.; Ware, J.; Ioannidis, J.
  • Preregistration is hard, and worthwhile
    Nosek, B. A.; Beck, E. D.; Campbell, L.; Flake, J. K.; Hardwicke, T. E.; Mellor, D. T.; van’t Veer, A. E.; Vazire, S.
  • Preregistration becoming the norm in psychological science
    Nosek, B. A.; Lindsay, D. S.
  • A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use
    Parry, D. A.; Davidson, B. I.; Sewall, C. J.; Fisher, J. T.; Mieczkowski, H.; Quintana, D. S.
  • Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?
    Pashler, H.; Wagenmakers, E.-J.
  • Many analysts and few incentives
    Ross, R. M.; Sulik, J.; Buczny, J.; Schivinski, B.
  • A survey on how preregistration affects the research workflow: Better science but more work
    Sarafoglou, A.; Kovacs, M.; Bakos, B.; Wagenmakers, E. J.; Aczel, B.
  • Bayesian estimation of finite population parameters in categorical data models incorporating order restrictions
    Sedransk, J.; Monahan, J.; Chiu, H.
  • Crowdsourced research: Many hands make tight work
    Silberzahn, R.; Uhlmann, E. L.
  • Pre-registration in social psychology: A discussion and suggested template
    van ’t Veer, A. E.; Giner-Sorolla, R
  • An agenda for purely confirmatory research
    Wagenmakers, E.-J.; Wetzels, R.; Borsboom, D.; van der Maas, H.; Kievit, R.
  • Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking
    Wicherts, J. M.; Veldkamp, C. L. S.; Augusteijn, H. E. M.; Bakker, M.; van Aert, R. C. M.; van Assen, M. A. L. M
  • Welcome to the tidyverse
    Wickham, H.; Averick, M.; Bryan, J.; Chang, W.; McGowan, L. D.; François, R.; Grolemund, G.; Hayes, A.; Henry, L.; Hester, J.; Kuhn, M.; Pedersen, T. L.; Miller, E.; Bache, S. M.; Müller, K.; Ooms, J.; Robinson, D.; Seidel, D. P.; Spinu, V.; . . . Yutani, H.