Collaborative Research: OAC Core: Small: Efficient and Policy-driven Burst Buffer Sharing

合作研究：OAC Core：小型：高效且策略驱动的突发缓冲区共享

• 批准号: 2008388
• 负责人: Zhao Zhang
• 金额: $ 29.29万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008388&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Small

Modern scientific research heavily relies on supercomputers. Supercomputing applications, such as traditional numerical simulations (HPC), data intensive applications (Big Data), and most recently, deep learning (DL) applications, are increasingly run on supercomputers to obtain timely results and explore new research methods that combine multiple application types. However, a bottleneck in their design reduces the potential performance of modern supercomputers. This project, bbThemis, addresses this problem by enabling efficient and policy-driven sharing of an intermediate storage layer known as a "burst buffer", so that more scientists and applications can leverage state-of-the-art storage techniques to significantly reduce their runtime and enhance the productivity of their research. This project will deliver substantial gains to almost every research area that uses HPC resources, leading to improved science and engineering methods and products in all fields. This research will have an immediate and significant impact on existing scientific applications and on deriving guidelines for next-generation HPC system design, deployment, and utilization. The project will also contribute to educational outcomes. In addition to students working directly on project goals, results developed in the project will be used in tutorial and training sessions at Texas Advanced Computing Center’s summer institute in deep learning and other major conferences, and in University of Illinois Urbana-Champaign student projects. The project is aligned with the National Strategic Computing Initiative (NSCI) to advance US leadership in HPC.This project, bbThemis (https://github.com/bbThemis), leverages a suite of technologies, such as disassociation of I/O processing from control logic, time-sliced intra I/O node sharing, function interception for low overhead POSIX I/O, and metadata and data placement for optimal individual application performance. It is investigating how to best apply these technologies, by: 1) Identifying optimal burst buffer configurations for a suite of representative supercomputing applications; 2) Proposing, prototyping, and verifying different design options to address intra-node and inter-node I/O performance sharing; and 3) Designing and evaluating a set of sharing policies, such as fair sharing and priority sharing, with real applications and I/O traces. This project will dramatically increase the sharing capacity of existing burst buffers and enhance domain scientists’ productivity at a large scale. It explores various sharing policies that permit efficient sharing of I/O resources and that meet the requirements of computing centers. The results will enable the provisioning of I/O resources, where users can request specific IOPS or bandwidth for a period of time. The prototype burst buffer sharing framework will immediately increase the capacity of existing supercomputers with enhanced I/O performance. The lessons learned will guide next-generation I/O system design for large scale systems. The general improvement of HPC, Big Data, and DL applications will also increase the coherence of the hardware and software used for data analytics computing and modeling and simulation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代科学研究在很大程度上依赖超级计算机。超级计算应用程序，例如传统数值模拟（HPC），数据密集型应用程序（大数据）以及最近的深度学习（DL）申请，越来越多地在超级计算机上运行，​​以获得及时的结果并探索结合多种应用程序类型的新研究方法。但是，设计中的瓶颈可降低现代超级计算机的潜在性能。该项目BBThemis通过实现称为“爆发缓冲液”的中间存储层的高效和政策驱动的共享来解决这个问题，以便更多的科学家和应用可以利用最先进的存储技术来显着降低其运行时并增强其研究产品。该项目将为几乎每个使用HPC资源的研究领域带来可观的收益，从而改善了所有领域的科学和工程方法和产品。这项研究将对现有的科学应用以及推导下一代HPC系统设计，部署和利用的指南产生直接和重大的影响。该项目还将有助于教育成果。除了直接从事项目目标的学生外，该项目中开发的结果还将在得克萨斯州高级计算中心的夏季研究所的教程和培训课程中使用，并在深度学习和其他主要会议上以及伊利诺伊大学Urbana-Champaign学生项目中使用。该项目与国家战略计算计划（NSCI）保持一致，以提高我们在HPC的领导地位，该项目，BBTHEMIS（https://github.com/bbthemis），利用一套技术套件，以i/o处理诸如控制逻辑，时间固定的Intera in Intera i/o i/o i/o OO OO OO OO OO OO OME SERIIST，诸如I/O处理之类的技术，例如元数据和数据放置，以实现最佳的个人应用程序性能。它正在研究如何最好地应用这些技术，作者：1）为一组代表性的超级计算应用程序识别最佳的突发缓冲配置； 2）提出，原型和验证不同的设计选项，以解决节点内和节点I/O性能共享； 3）设计和评估一组共享策略，例如公平共享和优先共享，与真实的应用程序和I/O跟踪。该项目将大大提高现有爆发缓冲区的共享能力，并在大规模上提高域科学家的生产率。它探讨了各种共享政策，这些政策允许有效地共享I/O资源，并满足计算中心的要求。结果将使I/O资源提供配置，用户可以在一段时间内请求特定的IOPS或带宽。原型爆发缓冲区共享框架将立即通过增强的I/O性能提高现有超级计算机的容量。所学的教训将指导大型系统的下一代I/O系统设计。 HPC，大数据和DL应用程序的总体改进还将提高用于数据分析计算和建模和模拟的硬件和软件的连贯性。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的审查标准来通过评估来获得支持的。

项目成果

Fine-Grained Policy-Driven I/O Sharing for Burst Buffers

• DOI: 10.1145/3581784.3607041
• 发表时间: 2023-06
• 期刊: SC23: International Conference for High Performance Computing, Networking, Storage and Analysis
• 作者: E. Karrels;Lei Huang;Yuhong Kan;Ishank Arora;Yinzhi Wang;D. Katz;W. Gropp;Zhao Zhang