Conference on Reproducibility and Replicability in Economics and Social Sciences (CRRESS)

A series of virtual and in-person panels on reproducibility, replicability, and transparency in the social sciences.

Funding:

NSF-2217493

Conference on Reproducibility and Replicability in Economics and Social Sciences (CRRESS)

Project Information

Authors: Aleksandr Michuda (Swarthmore College) and Lars Vilhuber (Cornell University)

Summary

The Conference on Reproducibility and Replicability in Economics and the Social Sciences (CRRESS) was a series of virtual and in-person panels on the topics of reproducibility, replicability, and transparency in the social sciences. The purpose of scientific publishing is the dissemination of robust research findings, exposing them to the scrutiny of peers and other interested parties. Scientific articles should accurately and completely provide information on the origin and provenance of data and on the analytical and computational methods used. Yet in recent years, doubts about the adequacy of the information provided in scientific articles and their addenda have been voiced.

The conferences addressed various topics in this area: the initiation of research, the conduct of research, the preparation of research for publication, and the scrutiny after publication. Undergraduates, graduate students, and career researchers were able to learn about best practices for transparent, reproducible, and scientifically sound research in the social sciences. The materials produced during the conference series are permanently archived and freely available to all interested parties.

Outcomes

The conference series consisted of 10 webinar sessions held during the 2022-2023 academic year, with a total of 149 registered participants. Sessions covered topics including:

Institutional support for reproducibility verification
Reproducibility and ethics
Teaching reproducibility in undergraduate and graduate education
Reproducibility with confidential and proprietary data
Disciplinary differences in reproducibility requirements
Funder and institutional requirements for reproducible archives

All sessions were recorded and are available on YouTube and archived in Cornell University’s eCommons repository.

The conference resulted in multiple publications in the Harvard Data Science Review’s Special Section on Reproducibility and Replicability and the ongoing Reinforcing Reproducibility and Replicability column, edited by Lars Vilhuber.

Further Information

For complete details, including session information, recordings, publications, and references, see the full outcomes report.

Website: https://labordynamicsinstitute.github.io/crress/

Online Book: https://labordynamicsinstitute.github.io/crress-book/

Publications

All 25 publications funded by grant SES-2217493:

Code Review, Reproducibility, and Improving the Scholarly Record

Nicholas Ottone and Limor Peer

Harvard Data Science Review, Jul 2025

DOI URL OA? Bib

@article{ottone2025,
  author = {Ottone, Nicholas and Peer, Limor},
  doi = {10.1162/99608f92.f9d748d4},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  timestamp = {2025-09-07T21:37:19Z},
  title = {Code {{Review}}, {{Reproducibility}}, and {{Improving}} the {{Scholarly Record}}},
  url = {https://hdsr.mitpress.mit.edu/pub/suy0nbh5},
  urldate = {2025-09-07},
  volume = {7},
  year = {2025},
  month_numeric = {7}
}

We Should Do More Direct Replications in Science

Stuart Buck

Harvard Data Science Review, Aug 2024

DOI URL OA? Bib

@article{buck2024,
  author = {Buck, Stuart},
  doi = {10.1162/99608f92.4eccc443},
  file = {Full Text PDF:/home/vilhuber/Zotero/storage/67KR5K4H/Buck - 2024 - We Should Do More Direct Replications in Science.pdf:application/pdf},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = aug,
  number = {3},
  publisher = {The MIT Press},
  timestamp = {2024-08-01T12:31:24Z},
  title = {We {{Should Do More Direct Replications}} in {{Science}}},
  url = {https://hdsr.mitpress.mit.edu/pub/4brnqfcf/release/1},
  urldate = {2024-08-01},
  volume = {6},
  year = {2024},
  month_numeric = {8}
}

Introducing Reproducible Research Standards at the World Bank

Maria Jones

Harvard Data Science Review, Oct 2024

DOI URL OA? Bib

@article{jones2024a,
  author = {Jones, Maria},
  doi = {10.1162/99608f92.21328ce3},
  file = {Full Text PDF:/home/vilhuber/Zotero/storage/FC6N8YV8/Jones - 2024 - Introducing Reproducible Research Standards at the World Bank.pdf:application/pdf},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = oct,
  number = {4},
  timestamp = {2025-02-06T00:32:34Z},
  title = {Introducing {{Reproducible Research Standards}} at the {{World Bank}}},
  url = {https://hdsr.mitpress.mit.edu/pub/jn90uuur},
  urldate = {2025-02-06},
  volume = {6},
  year = {2024},
  month_numeric = {10}
}

Why and How We Share Reproducible Research at Yale University’s Institution for Social and Policy Studies

Limor Peer

Harvard Data Science Review, Jan 2024

DOI URL OA? Bib

@article{peer2024,
  author = {Peer, Limor},
  doi = {10.1162/99608f92.dca148ba},
  file = {Full Text:/home/vilhuber/Zotero/storage/CC4IFJKW/Peer - 2024 - Why and How We Share Reproducible Research at Yale.pdf:application/pdf},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jan,
  number = {1},
  timestamp = {2024-05-02T12:57:11Z},
  title = {Why and {{How We Share Reproducible Research}} at {{Yale University}}'s {{Institution}} for {{Social}} and {{Policy Studies}}},
  url = {https://hdsr.mitpress.mit.edu/pub/lvp2cuc2},
  urldate = {2024-05-02},
  volume = {6},
  year = {2024},
  month_numeric = {1}
}

The Role of Third-Party Verification in Research Reproducibility

Christophe Pérignon

Harvard Data Science Review, Apr 2024

DOI URL OA? Bib

@article{perignon2024,
  author = {P{\'e}rignon, Christophe},
  doi = {10.1162/99608f92.6d4bf9eb},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = apr,
  number = {2},
  timestamp = {2024-05-02T12:59:00Z},
  title = {The {{Role}} of {{Third-Party Verification}} in {{Research Reproducibility}}},
  url = {https://hdsr.mitpress.mit.edu/pub/nqwl2udq},
  urldate = {2024-05-02},
  volume = {6},
  year = {2024},
  month_numeric = {4}
}

“Yes We Can!”: A Practical Approach to Teaching Reproducibility to Undergraduates

Richard Ball

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{ball2023,
  author = {Ball, Richard},
  doi = {10.1162/99608f92.9e002f7b},
  file = {Ball_2023_“Yes We Can.pdf:/home/vilhuber/Zotero/storage/VDLM44J7/Ball_2023_“Yes We Can.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  shorttitle = {``{{Yes We Can}}!''},
  timestamp = {2023-08-02T12:24:12Z},
  title = {``{{Yes We Can}}!'': {{A Practical Approach}} to {{Teaching Reproducibility}} to {{Undergraduates}}},
  url = {https://hdsr.mitpress.mit.edu/pub/mza9ii75/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Publishing Replication Packages: Insights From the Federal Reserve Bank of Kansas City

Courtney R. Butler

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{butler2023,
  author = {Butler, Courtney R.},
  doi = {10.1162/99608f92.aba61304},
  file = {Butler_2023_Publishing Replication Packages.pdf:/home/vilhuber/Zotero/storage/L2DRR6SQ/Butler_2023_Publishing Replication Packages.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  shorttitle = {Publishing {{Replication Packages}}},
  timestamp = {2023-08-02T12:23:22Z},
  title = {Publishing {{Replication Packages}}: {{Insights From}} the {{Federal Reserve Bank}} of {{Kansas City}}},
  url = {https://hdsr.mitpress.mit.edu/pub/j0ny4jrg/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Disciplinary Support: Why Is Reproducibility Not Uniformly Required across Disciplines?

2023

@misc{crress-video-5,
  collaborator = {Weeden, Kim and Sinclair, Betsy and Hoynes, Hilary and Vilhuber, Lars and Vilhuber, Lars},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  publisher = {Labor Dynamics Institute},
  shorttitle = {Disciplinary Support},
  timestamp = {2024-01-18T19:49:17Z},
  title = {Disciplinary Support: Why Is Reproducibility Not Uniformly Required across Disciplines?},
  url = {https://doi.org/10.7298/PKWJ-GM89},
  urldate = {2023-10-19},
  year = {2023}
}

Reproducibility With Confidential Data: The Experience of BPLIM

Paulo Guimarães

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{guimaraes2023,
  author = {Guimar{\~a}es, Paulo},
  doi = {10.1162/99608f92.54a00239},
  file = {Guimarães_2023_Reproducibility With Confidential Data.pdf:/home/vilhuber/Zotero/storage/T6IY8SQL/Guimarães_2023_Reproducibility With Confidential Data.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  shorttitle = {Reproducibility {{With Confidential Data}}},
  timestamp = {2023-08-02T12:23:56Z},
  title = {Reproducibility {{With Confidential Data}}: {{The Experience}} of {{BPLIM}}},
  url = {https://hdsr.mitpress.mit.edu/pub/rl6maajr/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Should Funders Require Reproducible Archives?

2023

@misc{halbert2023,
  collaborator = {Halbert, Martin and Martinez, Sebastian and Buck, Stuart and Vilhuber, Lars},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  timestamp = {2024-03-04T22:37:46Z},
  title = {Should Funders Require Reproducible Archives?},
  url = {https://doi.org/10.7298/21b2-yt23},
  urldate = {2024-03-04},
  year = {2023}
}

Reproducibility and Confidential or Proprietary Data: Can It Be Done?

2023

@misc{horton2023,
  collaborator = {Horton, John and Guimar{\~a}es, Paulo and Vilhuber, Lars and Michuda, Aleksandr},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  shorttitle = {Reproducibility and Confidential or Proprietary Data},
  timestamp = {2024-03-04T22:35:56Z},
  title = {Reproducibility and Confidential or Proprietary Data: Can It Be Done?},
  url = {https://doi.org/10.7298/3mjp-3h26},
  urldate = {2024-03-04},
  year = {2023}
}

Reproducibility in Economics: Status and Update

Hilary Hoynes

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{hoynes2023,
  author = {Hoynes, Hilary},
  doi = {10.1162/99608f92.80a1b88b},
  file = {Hoynes_2023_Reproducibility in Economics.pdf:/home/vilhuber/Zotero/storage/P52YZJHM/Hoynes_2023_Reproducibility in Economics.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  shorttitle = {Reproducibility in {{Economics}}},
  timestamp = {2023-08-02T12:24:42Z},
  title = {Reproducibility in {{Economics}}: {{Status}} and {{Update}}},
  url = {https://hdsr.mitpress.mit.edu/pub/lr5qgw5t/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Open Data and Code at the Urban Institute

Graham MacDonald

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{macdonald2023,
  author = {MacDonald, Graham},
  doi = {10.1162/99608f92.a631dfc5},
  file = {MacDonald_2023_Open Data and Code at the Urban Institute.pdf:/home/vilhuber/Zotero/storage/JRTKW7MZ/MacDonald_2023_Open Data and Code at the Urban Institute.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  timestamp = {2023-08-02T12:23:40Z},
  title = {Open {{Data}} and {{Code}} at the {{Urban Institute}}},
  url = {https://hdsr.mitpress.mit.edu/pub/qhszdvhn/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Why Can or Should Research Institutions Publish Replication Packages?

2023

@misc{macdonald2023a,
  collaborator = {MacDonald, Graham and Peer, Limor and Butler, Courtney and Michuda, Aleksandr and Vilhuber, Lars},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  publisher = {Labor Dynamics Institute},
  timestamp = {2024-01-18T19:49:11Z},
  title = {Why Can or Should Research Institutions Publish Replication Packages?},
  url = {https://doi.org/10.7298/pntg-rw59},
  urldate = {2024-01-18},
  year = {2023}
}

Reproducibility, Confidentiality, and Open Data Mandates

2023

@misc{mcgrail2023,
  collaborator = {McGrail, Kimberly and Taylor, S. Martin and Lucas, Matthew and Connolly, Marie},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  timestamp = {2024-03-04T22:38:17Z},
  title = {Reproducibility, Confidentiality, and Open Data Mandates},
  url = {https://doi.org/10.7298/hcry-nz34},
  urldate = {2024-03-04},
  year = {2023}
}

Data Citations and Reproducibility in the Undergraduate Curriculum

Diego Mendez-Carbajo and Alejandro Dellachiesa

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{mendez-carbajo2023,
  author = {{Mendez-Carbajo}, Diego and Dellachiesa, Alejandro},
  doi = {10.1162/99608f92.c2835391},
  file = {Mendez-Carbajo_Dellachiesa_2023_Data Citations and Reproducibility in the Undergraduate Curriculum.pdf:/home/vilhuber/Zotero/storage/YJJBYSZA/Mendez-Carbajo_Dellachiesa_2023_Data Citations and Reproducibility in the Undergraduate Curriculum.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  timestamp = {2023-08-02T12:24:28Z},
  title = {Data {{Citations}} and {{Reproducibility}} in the {{Undergraduate Curriculum}}},
  url = {https://hdsr.mitpress.mit.edu/pub/ohexlllx/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Institutional Support: How Do Journal Reproducibility Verification Services Work?

2023

@misc{perignon2023a,
  collaborator = {Perignon, Christophe and Greiner, Ben and Christian, Thu-Mai and Connolly, Marie},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  shorttitle = {Institutional Support},
  timestamp = {2024-03-04T22:37:06Z},
  title = {Institutional Support: {{How}} Do Journal Reproducibility Verification Services Work?},
  url = {https://doi.org/10.7298/0g2q-d958},
  urldate = {2024-03-04},
  year = {2023}
}

The Integration of Reproducibility into Social Science Graduate Education.

2023

@misc{reif2023,
  collaborator = {Reif, Julian and Freese, Jeremy and Wasser, David and Connolly, Marie},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  timestamp = {2024-03-04T22:38:42Z},
  title = {The Integration of Reproducibility into Social Science Graduate Education.},
  url = {https://doi.org/10.7298/bj42-e619},
  urldate = {2024-03-04},
  year = {2023}
}

The Case for Data Archives at Journals

Timothy C. Salmon

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{salmon2023,
  author = {Salmon, Timothy C.},
  doi = {10.1162/99608f92.db2a2554},
  file = {Salmon_2023_The Case for Data Archives at Journals.pdf:/home/vilhuber/Zotero/storage/K3CBIQCF/Salmon_2023_The Case for Data Archives at Journals.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  timestamp = {2023-08-02T12:24:54Z},
  title = {The {{Case}} for {{Data Archives}} at {{Journals}}},
  url = {https://hdsr.mitpress.mit.edu/pub/z9fpx2js/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Reinforcing Reproducibility and Replicability: An Introduction

Lars Vilhuber, Ian Schmutte, Aleksandr Michuda, and 1 more author

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{vilhuber2023c,
  author = {Vilhuber, Lars and Schmutte, Ian and Michuda, Aleksandr and Connolly, Marie},
  doi = {10.1162/99608f92.9ba2bd43},
  file = {Vilhuber et al_2023_Reinforcing Reproducibility and Replicability.pdf:/home/vilhuber/Zotero/storage/NE52GLZR/Vilhuber et al_2023_Reinforcing Reproducibility and Replicability.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  shorttitle = {Reinforcing {{Reproducibility}} and {{Replicability}}},
  timestamp = {2023-08-02T12:23:01Z},
  title = {Reinforcing {{Reproducibility}} and {{Replicability}}: {{An Introduction}}},
  url = {https://hdsr.mitpress.mit.edu/pub/l8dmf3cm/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Crisis? What Crisis? Sociology’s Slow Progress Toward Scientific Transparency

Kim A. Weeden

Harvard Data Science Review, Oct 2023

DOI URL OA? Bib

@article{weeden2023,
  author = {Weeden, Kim A.},
  doi = {10.1162/99608f92.151c41e3},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = oct,
  number = {4},
  shorttitle = {Crisis?},
  timestamp = {2023-11-27T02:13:57Z},
  title = {Crisis? {{What Crisis}}? {{Sociology}}'s {{Slow Progress Toward Scientific Transparency}}},
  url = {https://hdsr.mitpress.mit.edu/pub/6oi846rx},
  urldate = {2023-11-27},
  volume = {5},
  year = {2023},
  month_numeric = {10}
}

Costs and Benefits of Reproducibility in Finance and Economics

Toni Whited

Harvard Data Science Review, Jul 2023

DOI URL OA? Bib

@article{whited2023,
  author = {Whited, Toni},
  doi = {10.1162/99608f92.63de8e58},
  file = {Whited_2023_Costs and Benefits of Reproducibility in Finance and Economics.pdf:/home/vilhuber/Zotero/storage/JBZSQIBG/Whited_2023_Costs and Benefits of Reproducibility in Finance and Economics.pdf:application/pdf},
  issn = {2644-2353, 688-8513},
  journal = {Harvard Data Science Review},
  langid = {english},
  month = jul,
  number = {3},
  timestamp = {2023-08-02T12:25:07Z},
  title = {Costs and {{Benefits}} of {{Reproducibility}} in {{Finance}} and {{Economics}}},
  url = {https://hdsr.mitpress.mit.edu/pub/mnhzk8gq/release/1},
  urldate = {2023-08-02},
  volume = {5},
  year = {2023},
  month_numeric = {7}
}

Institutional Support: Should Journals Verify Reproducibility?

2022

@misc{imbens2022,
  collaborator = {Imbens, Guido and Salmon, Tim and Whited, Toni and Vilhuber, Lars},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  shorttitle = {Institutional Support},
  timestamp = {2024-03-04T22:33:45Z},
  title = {Institutional Support: {{Should}} Journals Verify Reproducibility?},
  url = {https://doi.org/10.7298/992J-RF71},
  urldate = {2024-03-04},
  year = {2022}
}

Should Teaching Reproducibility Be a Part of Undergraduate Education or Curriculum?

2022

@misc{mendez-carbajo2022,
  collaborator = {{Mendez-Carbajo}, Diego and Ball, Richard and Vilhuber, Lars and Schmutte, Ian},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  publisher = {[object Object]},
  timestamp = {2024-05-03T01:54:20Z},
  title = {Should Teaching Reproducibility Be a Part of Undergraduate Education or Curriculum?},
  url = {https://doi.org/10.7298/KBZA-0K11},
  urldate = {2024-05-03},
  year = {2022}
}

Reproducibility and Ethics - IRBs and Beyond

2022

@misc{meyer2022,
  collaborator = {Meyer, Michelle N. and Swauger, Shea and Kopper, Sarah and Vilhuber, Lars},
  copyright = {Creative Commons Attribution 4.0 International},
  doi = {},
  langid = {english},
  timestamp = {2024-03-04T22:34:55Z},
  title = {Reproducibility and Ethics - {{IRBs}} and Beyond},
  url = {https://doi.org/10.7298/cvqw-v588},
  urldate = {2024-03-04},
  year = {2022}
}