Collaborative Research: NGS: Dynamic Runtime and Compilation Support for I/O-Intensive Applications

合作研究：NGS：I/O 密集型应用程序的动态运行时和编译支持

• 批准号: 0406341
• 负责人: Alok Choudhary
• 金额: $ 33.65万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2004
• 资助国家: 美国
• 起止时间: 2004-08-01 至 2008-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0406341&HistoricalAwards=false
• 关键词: Collaborative; Research; NGS; Dynamic; Runtime

Abstract, CNS-0406341Tera-scale high-performance computing has enabled scientists to tackle very large and computationally challenging problems, such as those found in the scientific computing domain. However, as computing scales to levels never seen before, it also becomes extremely data intensive and I/O intensive. Thus, I/O is becoming a major bottleneck, thereby slowing the expected pace of scientific discovery and analysis of data. Furthermore, in order to cope with larger problems and data sizes, models and applications are being designed to be dynamic in nature. That is, the applications are dynamic both in terms of their computation patterns as well as data access patterns. Due to the complexities of systems and applications, it is, therefore, very important to address research issues and develop dynamic techniques at the level of runtime systems and compilers to scale I/O in the right proportions. Technical Merit:The objectives of this project are to design and develop next generation software techniques to address the data, I/O, and storage bottlenecks for large-scale scientific applications. Particularly, this project aims to investigate dynamic runtime and compilation techniques for scalable I/O optimizations for largescale systems. Another important aspect will be to drive these optimizations by learning and characterizing performance of I/O and data accesses, and subsequently using those to develop rules that will be used by dynamic runtime and compilation systems to enable high-performance I/O. Current state-of-the-art compiler support for I/O-intensive applications is tremendously lacking. Runtime needs of many large-scale I/O-intensive applications can benefit a lot from a robust dynamic compilation and linking infrastructure. The specific objectives of this project are:. Developing an understanding of dynamically varying data access needs of I/O-intensive applications,. Capturing dynamic access patterns and application steering information at runtime within a metadata manager,. Designing and implementing dynamic compilation techniques based on the runtime access patternsand performance statistics collected by the metadata manager,. Designing and implementing a layout manager that collects storage format (layout) suggestions from multiple concurrently executing applications and determines the globally acceptable storage layouts for disk-resident and tape-resident data,. Designing and implementing a high-level, dynamic, easy-to-use I/O library that can be invoked by the dynamic compiler/linker,. Investigating what types of user-specified hints can be passed to the runtime system/compiler, andhow they can be incorporated to reduce the overheads associated with dynamic compilation,. Evaluating the performance of the developed dynamic compilation/linking infrastructure underrealistic I/O-intensive workloads and quantifying the runtime overheads associated with dynamiccompilation, and. Providing the developed infrastructure and experimental findings in the public domain, andincorporating the research findings into the undergraduate and graduate curriculum.

摘要，CNS-0406341TERA规模的高性能计算使科学家能够解决非常大的计算挑战性问题，例如在科学计算领域中发现的问题。 但是，随着计算量表达到以前从未见过的水平，它也将变得非常密集，并且I/O密集型。因此，I/O正在成为主要的瓶颈，从而减慢了科学发现和数据分析的预期速度。此外，为了应对更大的问题和数据尺寸，模型和应用程序本质上是动态的。也就是说，在其计算模式和数据访问模式方面，应用程序都是动态的。因此，由于系统和应用程序的复杂性，解决研究问题并在运行时系统和编译器级别开发动态技术非常重要，以正确比例扩展I/O。技术优点：该项目的目标是设计和开发下一代软件技术，以解决大型科学应用程序的数据，I/O和存储瓶颈。特别是，该项目旨在调查针对大刻度系统的可扩展I/O优化的动态运行时和编译技术。另一个重要方面将是通过学习和表征I/O和数据访问的性能来推动这些优化，然后使用这些优化来开发这些规则，这些规则将被动态运行时和汇编系统使用，以启用高性能I/O。目前缺乏对I/O密集型应用程序的最新编译器支持。许多大规模I/O密集型应用程序的运行时间需求可以从鲁棒的动态汇编和链接基础架构中受益匪浅。该项目的具体目标是：对I/O密集型应用程序的动态变化数据访问需求有了了解。在元数据管理器内的运行时捕获动态访问模式和应用程序转向信息。根据Metadata Manager收集的运行时访问模式统计信息设计和实施动态汇编技术。设计和实施一个布局管理器，该布局管理器从多个同时执行应用程序中收集存储格式（布局）建议，并确定全球可接受的存储布局，以供磁盘居住和磁带驻留数据。设计和实施一个可以由动态编译器/链接器调用的高级，动态，易于使用的I/O库。研究可以将哪些类型的用户指定提示传递给运行时系统/编译器，然后将它们合并到如何减少与动态编译相关的开销。评估开发的动态汇编/链接基础架构不现实的I/O密集型工作负载，并量化与动态COMPILATION相关的运行时开销和。在公共领域提供开发的基础设施和实验发现，并将研究结果纳入本科和研究生课程。