Estimating the Reproducibility of Experimental Philosophy
Cova F., Strickland B., Abatista A., Allard A., Andow J., Attie M., Beebe J., Berniūnas R., Boudesseul J., Colombo M., Cushman F., Diaz R., N’Djaye Nikolai van Dongen N., Dranseika V., Earp BD., Torres AG., Hannikainen I., Hernández-Conde JV., Hu W., Jaquet F., Khalifa K., Kim H., Kneer M., Knobe J., Kurthy M., Lantian A., Liao SY., Machery E., Moerenhout T., Mott C., Phelan M., Phillips J., Rambharose N., Reuter K., Romero F., Sousa P., Sprenger J., Thalabard E., Tobia K., Viciana H., Wilkenfeld D., Zhou X.
Responding to recent concerns about the reliability of the published literature in psychology and other disciplines, we formed the X-Phi Replicability Project (XRP) to estimate the reproducibility of experimental philosophy (osf.io/dvkpr). Drawing on a representative sample of 40 x-phi studies published between 2003 and 2015, we enlisted 20 research teams across 8 countries to conduct a high-quality replication of each study in order to compare the results to the original published findings. We found that x-phi studies – as represented in our sample – successfully replicated about 70% of the time. We discuss possible reasons for this relatively high replication rate in the field of experimental philosophy and offer suggestions for best research practices going forward.