CI-EN: Collaborative Research: An Experimental Infrastructure and a Database of Real Faults to Foster Reproducibility in Software Engineering Research

CI-EN：协作研究：实验基础设施和真实故障数据库，以促进软件工程研究的可重复性

• 批准号: 1822251
• 负责人: Michael Ernst
• 金额: $ 26.81万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1822251&HistoricalAwards=false
• 关键词: CI; EN; Collaborative; Research; Experimental

A core tenet of the scientific method is reproducibility of experiments. To reproduce an empirical study in software engineering research, it is necessary to use the same experimental subjects and equivalent experimental infrastructure. Reproducibility in software engineering research is an important topic, as is evident from the recent establishment of artifact evaluation committees and from publication of papers with conflicting results. Prior efforts to establish reusable artifacts for specific software engineering domains are based on artificial data, are too small, are not uniformly organized, and/or lack extensible infrastructure. This holds the software engineering research community back, hampers reproducibility and comparability, and reduces confidence in empirical results.This project promotes reproducibility and comparability in software engineering research, through a large-scale database of real software defects and a standardized infrastructure that enables controlled experiments. It enhances an existing prototype (Defects4J) by adding new artifacts, adding infrastructure for labeling and experimenting with the artifacts, and improving usability. Specifically, this project provides (1) the world's largest database of isolated and annotated real software defects, (2) an extensible mining infrastructure that supports automated defect mining and isolation, (3) an extensible experimental infrastructure that also includes a statistics and replication package component, and (4) tutorials and training materials for researchers and educators.This project advances software engineering research in three ways. (1) It enables researchers to perform more realistic experiments, using real instead of artificial defects. (2) It frees researchers from the burden of (re-)developing an experimental infrastructure, allowing them to focus on research ideas. (3) It fosters reproducibility and comparability by providing reusable artifacts and data sets. Furthermore, this project advances software engineering education at the undergraduate and graduate level: students can experiment with existing techniques, reproduce published results, or quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学方法的核心原则是实验的可重复性。为了重现软件工程研究中的经验研究，有必要使用相同的实验主题和同等的实验基础设施。软件工程研究中的可重复性是一个重要的主题，从最近成立的人工制品评估委员会以及出版有关结果相互矛盾的论文中可以明显看出。先前为特定软件工程领域建立可重复使用的工件的努力是基于人造数据，太小，不统一组织和/或缺乏可扩展的基础架构。这样可以使软件工程研究社区备份，妨碍了可重复性和可比性，并降低了对经验结果的信心。该项目通过大型的真实软件缺陷和标准化的基础架构来促进软件工程研究的可重复性和可比性，以实现控制实验。它通过添加新的工件，添加用于标记和实验工件的基础架构并提高可用性来增强现有原型（缺陷4J）。具体而言，该项目提供（1）世界上最大的孤立和注释实际软件缺陷的数据库，（2）可扩展的采矿基础设施，支持自动缺陷挖掘和隔离，（3）可扩展的实验基础设施，其中包括统计和重复套件的统计和重复研究的材料，以及研究人员和培训材料。 （1）它使研究人员能够使用真实的缺陷进行更现实的实验。 （2）它使研究人员摆脱了（重新）开发实验基础设施的负担，使他们能够专注于研究思想。 （3）通过提供可重复使用的工件和数据集来促进可重复性和可比性。 此外，该项目在本科和研究生层面上推进了软件工程教育：学生可以尝试现有技术，复制已发表的结果或快速评估新的想法。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估来通过评估来获得支持的审查审查审查标准。