Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole CR., Mathur MB., Baranski E., Bart-Plange DJ., Buttrick NR., Chartier CR., Corker KS., Corley M., Hartshorne JK., IJzerman H., Lazarević LB., Rabagliati H., Ropovik I., Aczel B., Aeschbach LF., Andrighetto L., Arnal JD., Arrow H., Babincak P., Bakos BE., Baník G., Baskin E., Belopavlović R., Bernstein MH., Białek M., Bloxsom NG., Bodroža B., Bonfiglio DBV., Boucher L., Brühlmann F., Brumbaugh CC., Casini E., Chen Y., Chiorri C., Chopik WJ., Christ O., Ciunci AM., Claypool HM., Coary S., Čolić MV., Collins WM., Curran PG., Day CR., Dering B., Dreber A., Edlund JE., Falcão F., Fedor A., Feinberg L., Ferguson IR., Ford M., Frank MC., Fryberger E., Garinther A., Gawryluk K., Ashbaugh K., Giacomantonio M., Giessner SR., Grahe JE., Guadagno RE., Hałasa E., Hancock PJB., Hilliard RA., Hüffmeier J., Hughes S., Idzikowska K., Inzlicht M., Jern A., Jiménez-Leal W., Johannesson M., Joy-Gaba JA., Kauff M., Kellier DJ., Kessinger G., Kidwell MC., Kimbrough AM., King JPJ., Kolb VS., Kołodziej S., Kovacs M., Krasuska K., Kraus S., Krueger LE., Kuchno K., Lage CA., Langford EV., Levitan CA., de Lima TJS., Lin H., Lins S., Loy JE., Manfredi D., Markiewicz Ł., Menon M., Mercier B., Metzger M., Meyet V., Millen AE., Miller JK., Montealegre A.
Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p