SI2-SSE: EASE: Improving Research Accountability through Artifact Evaluation

SI2-SSE：EASE：通过工件评估提高研究问责制

• 批准号: 1535232
• 负责人: Bruce Childers
• 金额: $ 49.95万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1535232&HistoricalAwards=false
• 关键词: SI2; SSE; EASE; Improving; Research

Research in computer systems, particularly in the first stages of creating a new innovation, relies almost exclusively on software prototypes, simulators, benchmarks, and data sets to understand the benefits and costs of new ideas in computers, ranging from consumer devices to exascale systems. These artifacts are used to evaluate new capabilities, algorithms, bottlenecks and trade-offs. Empirical study is behind the rapid pace of innovation in creating faster, lower energy and more reliable systems. This experimental approach lies at the core of development that fuels the nation's information economy. Given the critical importance of experimental study to developing new computer systems, several efforts are underway to curate experimental results through accountable research. One effort, Artifact Evaluation (AE), is being adopted to promote high quality artifacts and experimentation, including making public the experimental information necessary for reproducibility. However, the rapid adoption of AE is hampered by technical challenges that create a high barrier to the process: there is no consistent or simple environment, or mechanism, to package and reproduce experiments for AE. Authors rely on their own approaches, leading to much time consumed, as well as considerable variability in the ways materials are prepared and evaluated, unnecessarily obstructing the AE process. To overcome the technical challenges with AE, and to more broadly encourage adoption of AE in computer science and engineering research, this project is developing a software infrastructure, Experiment and Artifact System for Evaluation (EASE), to create and run experiments specifically for AE, in which authors create, conduct and share artifacts and experiments. It allows for repeating, modifying, and extending experiments. Authors may also use EASE to package and upload their experiments for archival storage in a digital library. EASE is being developed and deployed for two use cases, namely compilers and real-time systems, keeping the project tractable to address specific needs. These communities have overlapping but also distinct requirements, helping to ensure EASE can also be extended andused by other computer systems research communities as well.EASE will be release as open source software, based on an Experiment Management System (EMS) previously developed by the project investigator in a project call Open Curation for Computer Architecture Modeling (OCCAM), used to define and conduct experiments using computer architecture simulators. Using EMS as a starting point, EASE will provide AE support, by: 1) separating EMS from OCCAM's repository and hardware services, transforming the EMS infrastructure into EASE, a fully standalone, sustainable, and extensible platform for AE; 2) supporting record and replay (for repeating and reproducing results, as well as provenance) of artifacts and experiments as part of normal development and experimental practice to ease participation in AE by authors and evaluators; 3) supporting artifacts,workflows of artifacts and experiments that run directly on a machine, including specialized hardware and software, and run indirectly on a simulator or emulator; 4) allowing both user-level (artifacts and experiments as user processes) and system-level (artifacts and experiments involving kernel changes) innovations; 5) providing consistent/uniform access, whether locally or remotely, to artifacts and experiments; 6) simplifying viewing, running, modifying, and comparing experiments by innovators (i.e., during innovation development), artifact evaluators (during AE), and archive users (after publication); 7) enabling indexing (object locators and search tags) and packaging of artifacts and experiments for AE and for archival deployment (e.g., to ACM?s or IEEE?s Digital Library); and 8) refining, expanding, generalizing, and documenting EASE to ensure it is robust, maintainable and extensible, and that it can be used and sustained by different CSR communities (starting with real-time and compilers, given their different artifacts, data and methods).

计算机系统的研究，尤其是在创建新创新的第一阶段，几乎完全依赖软件原型，模拟器，基准和数据集，以了解计算机中新想法的收益和成本，从消费者设备到外部系统。这些文物用于评估新功能，算法，瓶颈和权衡。实证研究是创新快速发展的速度，以创造更快，较低的能源和更可靠的系统。这种实验方法在于为国家信息经济增添的发展核心。 鉴于实验研究对开发新计算机系统的重要性至关重要，通过负责研究正在进行一些努力来策划实验结果。 正在采用一项努力，即工件评估（AE），以促进高质量的人工制品和实验，包括公开可重复性所需的实验信息。但是，AE的迅速采用受到技术挑战的阻碍，这些挑战构成了该过程的高障碍：没有一致或简单的环境或机制来打包和重现AE的实验。作者依靠自己的方法，导致了很多时间消耗，并且在准备和评估材料的方式上，不必要地妨碍了AE流程的方式。 为了克服AE的技术挑战，并更广泛地鼓励AE在计算机科学和工程研究中采用AE，该项目正在开发一个软件基础架构，实验和工件评估系统（EASE），以创建和运行专门针对AE的实验，在该实验中，作者创建，进行和共享艺术品和实验。它允许重复，修改和扩展实验。作者还可以轻松打包和上传其实验，以在数字库中进行档案存储。正在开发和部署两种用例，即编译器和实时系统，使项目可以解决特定需求。这些社区具有重叠但也有不同的要求，有助于确保其他计算机系统研究社区也可以扩展并使用。ease将根据项目调查人员在项目调用中以实验管理系统（EMS）开发的开源软件（EMS）来释放，以定义和进行计算机架构师的计算机架构模型（OCCAM）开放策划（OCCAM）。以：1）将EMS与OCCAM的存储库和硬件服务分开，将EMS基础架构转换为Ase，是一个完全独立的，可持续的，可扩展的AE平台； 2）支持记录和重播（用于重复和复制结果，以及出处）和实验，这是正常发展和实验实践的一部分，以缓解作者和评估者参与AE； 3）支撑直接在机器上运行的工件和实验的工件，包括专业的硬件和软件，并在模拟器或仿真器上间接运行； 4）允许用户级（作为用户流程的工件和实验）和系统级别（涉及内核变化的工件和实验）创新的创新； 5）提供一致/统一的访问，无论是在本地还是远程访问，都可以访问工件和实验； 6）简化创新者（即创新开发期间），工件评估人员（在AE期间）和档案用户（出版后）的创新者（即，在创新过程中）进行了观看，运行，修改和比较； 7）启用索引（对象定位器和搜索标签），以及用于AE和档案部署的工件和实验的包装（例如，到ACM？s或ieEEE？s数字库）； 8）精炼，扩展，概括和记录易度性，以确保其坚固，可维护和可扩展，并可以由不同的CSR社区使用和维持（鉴于其不同的工件，数据和方法，从实时和编译器开始）。