Testing Modality
The vast majority of the papers included in our study were originally run in university labs. Ideally, replication would therefore also have taken place in the lab. Of course, our project perfectly coincided with a global pandemic that rendered lab studies unsafe for well over a year. At the outset of the project, it was unclear how long the pandemic would last. In order to begin work, the replication team developed the following strategy based on how the data was collected in each original study.
Original study run virtually and asynchronously
The replication was also run virtually and asynchronously which could begin prior to labs reopening. A failure to replicate did not trigger any additional testing.
Studies: Schweitzer & Cachon (2000)
Original study run on Amazon Mechanical Turk
The replication was also run on MTurk which could begin prior to labs reopening. A failure to replicate did not trigger any additional testing.
Studies: Shunko, et al. (2018)
Original study run in person and is an individual decision task
In order to progress the project, we began by attempting replication virtually and asynchronously using a university subject pool. The original study materials were modified if needed to allow for online testing. Given that the replication environment is substantially different from a controlled lab, failure to replicate virtually triggered an additional attempt at replication in a face-to-face laboratory setting once labs reopened. In such cases, we report all data and results but defer to the lab results for any aggregate claims about replicability.
Studies: Chen, et al. (2013); Davis, et al. (2011); Engelbrecht-Wiggans & Katok (2008); Kremer, et al. (2011)
Original study run in person and involves interaction among participants
We deferred these studies until labs reopened. The replications took place in a traditional, face-to-face controlled lab environment. A failure to replicate did not trigger any additional testing.
Studies: Croson & Donohue (2006); Ho & Zhang (2008); Kremer & Debo (2016); Ozer, et al. (2011)
Power Analysis and Target Sample Sizes
The objective of this project is to establish the robustness of experimental findings through a large-scale, high-powered replication. To this end, we established the following three criteria for establishing sample size. Each study should…
Achieve at least 90% power
Using the effect size and sample size from the original study, we computed the required sample size to achieve 90% power. The sample size for each replication at each site must be meet this minimum threshold.
Be at least as large as the original study
Each of our replication attempts will be as large or larger than the original study. In this way, our results will be at least as informative as the original findings. The sample size for each replication at each site must be meet this minimum threshold.
Have at least 40 participants
In some rare cases, the above two criteria were both satisfied with quite small samples. In response, the replication team decided to impose an absolute minimum sample size. The sample size for each replication at each site must be meet this minimum threshold.
Guiding Principles
We have attempted to list some of the principles that guided our decision making throughout this project.
Offer Transparency into Process and Access to Materials and Results
We have posted raw data and analysis files (in Stata or R) for each replication study as well as for all tables and figures that appear in the summary paper. Pending permission of the original authors, we intend to add additional materials such as full instructions, experimental software, etc.
Providing an Opportunity for Original Authors to Respond
We shared our results and post-replication reports with the original authors prior to publication. All original authors were given the opportunity to provide a written response to be posed on this website and published in the appendix of the summary paper.
Fidelity to Original Study
Our initial goal was to perform an exact replication (same modality and materials as original), but this was not always possible due to the pandemic, the loss of original materials, etc. We have done our best to disclose deviations from the original study (whether planned or unplanned).