The Management Science Replication Project is designed to solidly establish (or not) key findings of highly impactful experimental papers in the journal Management Science through high-powered replication across multiple locations.
Project Completed
A Replication Study of Operations Management Experiments in Management Science was published online by Management Science on July 11, 2023. A comprehensive set of supplemental materials (~90MB) is available on the Management Science website and includes experimental stimuli, data collected, and analysis files for the ten papers included as well as analysis files for the main paper. The paper is also available on SSRN.
Related News
The paper was the lead article in the September 2023 issue of Management Science. David Simchi-Levi posted an article to the From the Editor blog entitled Behavioral science’s credibility is at risk. Replication studies can help which contains supportive commentary from Colin F. Camerer (California Institute of Technology) and Yan Chen (University of Michigan) on September 1, 2023.
The project received a Christiaan Huygens Reproduction and Replication Prize from the Institute for Replication. The award was announced on X (formerly Twitter) on October 23, 2023.
The paper was chosen as a featured Management Science article, and the authors were invited to write a short piece for the Management Science Review blog. The post, entitled The Value of Replications in Operations Management, briefly describes the project and comments on its importance in light of recent developments concerning the credibility of academic research. It appeared on November 7, 2023.
The paper received the 2024 Management Science Best Paper Award in Operations Management from the MSOM Society Board on July 1, 2024. The award is given to the manuscript judged to be most deserving for its contribution to the theory and practice of operations management among all operations papers published in the past 3 years at Management Science.
Origin
In his From the Editor blog post in January 2020, David Simchi-Levi, editor-in-chief of Management Science, issued the following challenge to the community:
The editorial board would like to publish a paper, likely a Fast Track paper, that reports replicability of laboratory experiments published by Management Science. This was done in economics, see Camerer et al. (2016), and in social science, see Camerer et al. (2018), and it is time to do the same for Management Science papers.
In the subsequent months, we formed a project team of eight scholars from five universities to respond. This project team was formally announced in April 2020 in a From the Editor blog post entitled A Wish and A Challenge.
Procedure
- Establish list of potentially replicable papers. The team identified a list experimental papers published in Management Science between 2000 and 2020 with fairly standard subject pools and at least one statistically significant treatment effect. This included twenty-four papers in five main categories: inventory management; supply chain and contracting; queueing and production; forecasting; and sourcing and procurement.
- Narrow list based on community feedback. On September 4, 2020, Elena Katok posted a message entitled Management Science Replication Project to the Behavioral Operations Management and MSOM Society sections on INFORMS Connect asking community members to vote on which papers they would like to see replicated. For archival purposes, we have left the survey available in its original form here, but votes will no longer be considered. On October 26, 2020, we tallied the votes of the roughly one hundred respondents and arrived at a final list of ten papers.
- Assign papers to replication sites. For an added layer of robustness, the team decided to perform each replication at two different study sites. One site was designated the “primary” site. Our project team page details these assignments.
- Establish replication protocol. In concert with the original study authors, the primary site team member(s) established one or two specific findings for replication, conducted a power analysis to establish required sample size, and detailed all other testing procedures. The primary team filed a pre-registration on aspredicted.org for each paper and prepared a pre-replication report. Both of these documents are available on the individual paper pages listed at right.
- Ask community to predict likelihood of replication. A survey is currently being prepared. Once ready, it will be posted the Behavioral Operations Management and MSOM Society sections on INFORMS Connect.
- Perform replication studies at both sites.
- Compile results in a summary paper and submit for publication. All replication data will be posted to this website. We will prepare a detailed post-replication report for each of the ten chosen papers.