Collaborative Research: Disciplinary Improvements: Repeto: Building a Network for Practical Reproducibility in Experimental Computer Science

协作研究：学科改进：Repeto：构建实验计算机科学实用可重复性网络

• 批准号: 2226407
• 负责人: Cormac Flanagan
• 金额: $ 92.99万
• 依托单位: University of California-Santa Cruz
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2226407&HistoricalAwards=false
• 关键词: Collaborative; Research; Disciplinary; Improvements; Repeto

The Repeto project will foster community practices to make reproducibility a part of mainstream research and education activities in computer science. The project seeks to understand the cost/benefit equation of reproducibility for the computer science systems community, the factors that make reproducibility feasible or infeasible, as well as isolate factors (be they technical or usage oriented) that make practical reproducibility of experiments difficult. This research coordination network will develop a range of activities from teaching methodology for packaging experiments for cost-effective replication; using reproducibility in teaching; collaboration with reproducibility initiatives sponsored through conferences and institutions; community events emphasizing repeating or replicating experiments such as hackathons, competitions, or rankings; fostering repositories of replicable experiments and monitoring their usage/replication; to reporting on state of art and emergent requirements for the support of practical reproducibility. The outcomes of the proposal will be a collection of computer science experiments replicable on open platforms, an understanding of how much and to what extent they are used in mainstream research and education activities via relevant metrics, as well as a series of reports on current enablers and obstacles towards mainstream use of reproducibility in computer science research.Replicable experiments will be created using platform programmability tools including the Chameleon environment and associated software such as CHI, Trovi, and Jupyter notebooks. This platform programmability approach allows experimenters to express complex experimental topologies in repeatable and persistent ways. Combining platform programmability with executable notebooks will allow investigators to capture the full experimental process for subsequent replication by other researchers.This award by the CISE Office of Advanced Cyberinfrastructure is jointly supported by the CISE Computer and Networked Systems Division.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Repeto 项目将促进社区实践，使可重复性成为计算机科学主流研究和教育活动的一部分。 该项目旨在了解计算机科学系统界的可重复性的成本/效益方程、使可重复性可行或不可行的因素，以及使实验的实际可重复性变得困难的隔离因素（无论是技术性的还是面向使用的）。 该研究协调网络将开展一系列活动，从教学方法到包装实验以实现经济有效的复制；在教学中使用可重复性；与会议和机构赞助的可重复性倡议合作；强调重复或复制实验的社区活动，例如黑客马拉松、竞赛或排名；培育可复制实验的存储库并监测其使用/复制；报告支持实际可重复性的最新技术和紧急要求。该提案的成果将是一系列可在开放平台上复制的计算机科学实验，通过相关指标了解它们在主流研究和教育活动中的使用程度和程度，以及关于当前推动因素的一系列报告计算机科学研究中主流使用可重复性的障碍。可重复的实验将使用平台可编程工具创建，包括 Chameleon 环境和相关软件，例如 CHI、Trovi 和 Jupyter 笔记本。 这种平台可编程性方法允许实验者以可重复和持久的方式表达复杂的实验拓扑。将平台可编程性与可执行笔记本相结合将使研究人员能够捕获完整的实验过程，以便其他研究人员随后进行复制。该奖项由 CISE 高级网络基础设施办公室颁发，并得到 CISE 计算机和网络系统部门的共同支持。该奖项反映了 NSF 的法定使命和通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。