SI2-SSE: EASE: Improving Research Accountability through Artifact Evaluation

SI2-SSE：EASE：通过工件评估提高研究问责制

• 批准号: 1535232
• 负责人: Bruce Childers
• 金额: $ 49.95万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1535232&HistoricalAwards=false
• 关键词: SI2; SSE; EASE; Improving; Research

Research in computer systems, particularly in the first stages of creating a new innovation, relies almost exclusively on software prototypes, simulators, benchmarks, and data sets to understand the benefits and costs of new ideas in computers, ranging from consumer devices to exascale systems. These artifacts are used to evaluate new capabilities, algorithms, bottlenecks and trade-offs. Empirical study is behind the rapid pace of innovation in creating faster, lower energy and more reliable systems. This experimental approach lies at the core of development that fuels the nation's information economy. Given the critical importance of experimental study to developing new computer systems, several efforts are underway to curate experimental results through accountable research. One effort, Artifact Evaluation (AE), is being adopted to promote high quality artifacts and experimentation, including making public the experimental information necessary for reproducibility. However, the rapid adoption of AE is hampered by technical challenges that create a high barrier to the process: there is no consistent or simple environment, or mechanism, to package and reproduce experiments for AE. Authors rely on their own approaches, leading to much time consumed, as well as considerable variability in the ways materials are prepared and evaluated, unnecessarily obstructing the AE process. To overcome the technical challenges with AE, and to more broadly encourage adoption of AE in computer science and engineering research, this project is developing a software infrastructure, Experiment and Artifact System for Evaluation (EASE), to create and run experiments specifically for AE, in which authors create, conduct and share artifacts and experiments. It allows for repeating, modifying, and extending experiments. Authors may also use EASE to package and upload their experiments for archival storage in a digital library. EASE is being developed and deployed for two use cases, namely compilers and real-time systems, keeping the project tractable to address specific needs. These communities have overlapping but also distinct requirements, helping to ensure EASE can also be extended andused by other computer systems research communities as well.EASE will be release as open source software, based on an Experiment Management System (EMS) previously developed by the project investigator in a project call Open Curation for Computer Architecture Modeling (OCCAM), used to define and conduct experiments using computer architecture simulators. Using EMS as a starting point, EASE will provide AE support, by: 1) separating EMS from OCCAM's repository and hardware services, transforming the EMS infrastructure into EASE, a fully standalone, sustainable, and extensible platform for AE; 2) supporting record and replay (for repeating and reproducing results, as well as provenance) of artifacts and experiments as part of normal development and experimental practice to ease participation in AE by authors and evaluators; 3) supporting artifacts,workflows of artifacts and experiments that run directly on a machine, including specialized hardware and software, and run indirectly on a simulator or emulator; 4) allowing both user-level (artifacts and experiments as user processes) and system-level (artifacts and experiments involving kernel changes) innovations; 5) providing consistent/uniform access, whether locally or remotely, to artifacts and experiments; 6) simplifying viewing, running, modifying, and comparing experiments by innovators (i.e., during innovation development), artifact evaluators (during AE), and archive users (after publication); 7) enabling indexing (object locators and search tags) and packaging of artifacts and experiments for AE and for archival deployment (e.g., to ACM?s or IEEE?s Digital Library); and 8) refining, expanding, generalizing, and documenting EASE to ensure it is robust, maintainable and extensible, and that it can be used and sustained by different CSR communities (starting with real-time and compilers, given their different artifacts, data and methods).

计算机系统的研究，特别是在创造新创新的第一阶段，几乎完全依赖于软件原型、模拟器、基准测试和数据集来了解计算机新想法的好处和成本，范围从消费设备到百亿亿级系统。这些工件用于评估新功能、算法、瓶颈和权衡。实证研究是创造更快、更低能耗和更可靠系统的快速创新背后的动力。这种实验方法是推动国家信息经济发展的核心。 鉴于实验研究对于开发新计算机系统的至关重要性，人们正在努力通过负责任的研究来整理实验结果。 其中一项工作是工件评估（AE），旨在促进高质量的工件和实验，包括公开可重复性所需的实验信息。然而，AE 的快速采用受到技术挑战的阻碍，这些技术挑战给该过程带来了很高的障碍：没有一致或简单的环境或机制来打包和重现 AE 实验。作者依赖自己的方法，导致耗费大量时间，并且材料准备和评估的方式存在很大差异，不必要地阻碍了 AE 过程。 为了克服 AE 的技术挑战，并更广泛地鼓励在计算机科学和工程研究中采用 AE，该项目正在开发软件基础设施、实验和工件评估系统 (EASE)，以专门为 AE 创建和运行实验，作者在其中创建、进行和分享工件和实验。它允许重复、修改和扩展实验。作者还可以使用 EASE 打包并上传他们的实验，以便在数字图书馆中存档存储。 EASE 正在针对两个用例（即编译器和实时系统）进行开发和部署，从而使项目易于处理以满足特定需求。这些社区有重叠但又不同的要求，有助于确保 EASE 也可以被其他计算机系统研究社区扩展和使用。EASE 将作为开源软件发布，基于该项目先前开发的实验管理系统 (EMS)计算机架构建模开放管理 (OCCAM) 项目的研究员，该项目用于使用计算机架构模拟器定义和进行实验。以EMS为起点，EASE将通过以下方式提供AE支持：1）将EMS与OCCAM的存储库和硬件服务分离，将EMS基础设施转变为EASE，一个完全独立、可持续和可扩展的AE平台； 2) 支持工件和实验的记录和重放（用于重复和再现结果以及出处），作为正常开发和实验实践的一部分，以方便作者和评估者参与 AE； 3）支持直接在机器上运行的工件、工件的工作流程和实验，包括专用硬件和软件，以及间接在模拟器或仿真器上运行； 4）允许用户级（作为用户进程的工件和实验）和系统级（涉及内核更改的工件和实验）创新； 5）提供对工件和实验的一致/统一访问，无论是本地还是远程； 6) 简化创新者（即在创新开发期间）、工件评估者（在 AE 期间）和存档用户（在发布后）的查看、运行、修改和比较实验； 7) 启用索引（对象定位器和搜索标签）以及对 AE 和存档部署的工件和实验进行打包（例如，到 ACM 或 IEEE 数字图书馆）； 8) 完善、扩展、概括和记录 EASE，以确保它的健壮性、可维护性和可扩展性，并且可以由不同的 CSR 社区使用和维护（从实时和编译器开始，考虑到它们不同的工件、数据和方法）。