NSCI Elements: Software - PFSTRASE - A Parallel FileSystem TRacing and Analysis SErvice to Enhance Cyberinfrastructure Performance and Reliability

NSCI Elements：软件 - PFSTRASE - 用于增强网络基础设施性能和可靠性的并行文件系统跟踪和分析服务

• 批准号: 1835135
• 负责人: Richard Evans
• 金额: $ 38.59万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1835135&HistoricalAwards=false
• 关键词: NSCI; Elements; Software; PFSTRASE; Parallel

This project will develop an open-source software service, the Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE), that improves the reliability and performance of data storage systems for the nation?s largest supercomputers. As simulations and computations represent reality more faithfully they grow commensurately in scale along with the size of the data they consume and generate. To handle the storage and movement of this data, supercomputing systems are built on the backbone of massively parallel data storage systems. Due to their parallel nature these storage systems are capable of moving data at hundreds of times the speed of conventional storage systems, enabling otherwise impractical computations. The performance capabilities these storage systems provide is accompanied by a complexity that results in them often functioning significantly less than optimally and even in some instances failing. This results in wasted computational time and ultimately lost scientific progress. The state of development of tools that could cast light on these problems and improve storage system reliability and performance is inadequate for current and future computing systems. PFSTRASE will fill this gap by continually and automatically monitoring storage system health and performance, providing insights through an easy to use interface that will improve the reliability and performance of storage and supercomputer systems. Parallel filesystems (PFSs) are the most critical high-availability components of High Performance Computing (HPC) architectures, providing input/output (I/O) services to running computations, the environment that users and system services operate in, and storage for applications and data. Because of this central role, failure or performance degradation events in the PFS impact every user of an HPC resource. PFS events must be dealt with quickly and effectively by system administrators; however, there is typically insufficient information to establish precise causal relationships between PFS activity and events, impeding the implementation of timely and targeted remedies. To fill this information gap, an open-source Parallel FileSystem TRacing and Analysis SErvice (PFSTRASE) that traces and analyzes the requisite data to establish causal relationships between PFS activity and both realized and imminent events will be developed. This project will implement the service for the open-source Lustre filesystem, which is the most commonly used PFS at large-scale HPC sites. Loads for specific PFS directory and file operations will be measured and incorporated into the service to construct authentic server load contributions from every job, process, and user. The service?s infrastructure will continuously monitor the entire PFS and generate a real-time, seamless representation that connects contributions of jobs, processes, and users to storage server loads, network bandwidth, and storage capacities. The infrastructure will provide an easily navigable web interface that presents this data, both real-time and historical, in a visual format.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将开发开源软件服务，即并行文件系统跟踪和分析服务（PFSTRASE），该服务可改善国家最大的超级计算机的数据存储系统的可靠性和性能。 随着模拟和计算更忠实地代表了现实，它们以及它们消耗和生成的数据的大小都在规模上相称。 为了处理这些数据的存储和移动，超级计算系统建立在大量并行数据存储系统的骨干上。 由于其平行性，这些存储系统能够在传统存储系统的速度上移动数据，从而实现了不切实际的计算。 这些存储系统提供的性能功能伴随着一种复杂性，使它们的功能通常明显少于最佳功能，甚至在某些情况下失败。 这导致了浪费的计算时间，并最终失去了科学进步。 可以揭示这些问题并提高存储系统可靠性和性能的工具的开发状态对于当前和将来的计算系统不足。 PFSTRASE将通过不断，自动监视存储系统的健康和性能来填补这一空白，从而通过易于使用的接口提供见解，从而提高存储和超级计算机系统的可靠性和性能。并行文件系统（PFSS）是高性能计算（HPC）体系结构的最关键的高可用性组件，为运行计算提供输入/输出（I/O）服务，用户和系统服务在其中运行的环境以及用于应用程序和数据的存储。由于这种核心角色，PFS中的失败或性能降解事件会影响HPC资源的每个用户。系统管理员必须快速有效地处理PFS事件；但是，通常没有足够的信息来建立PFS活动与事件之间的精确因果关系，从而阻碍了及时和有针对性的补救措施的实施。为了填补此信息差距，将开发一个必要的数据，以建立PFS活动与已实现和迫在眉睫的事件之间建立因果关系的开源并行文件系统跟踪和分析服务（PFSTRASE）。该项目将为开源光泽文件系统实施服务，该系统是大规模HPC站点上最常用的PFS。特定PFS目录和文件操作的负载将被测量并合并到服务中，以构建每个作业，过程和用户​​的真实服务器负载贡献。该服务的基础架构将不断监视整个PFS，并生成实时的无缝表示形式，该表示将作业，流程和用户的贡献连接到存储服务器负载，网络带宽和存储容量。基础架构将提供一个易于导航的Web界面，以视觉格式呈现这些数据，无论是实时还是历史的。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准的评估值得支持的。

项目成果

Democratizing Parallel Filesystem Monitoring

并行文件系统监控民主化

• DOI: 10.1109/cluster49012.2020.00065
• 发表时间: 2020
• 期刊: 2020 IEEE International Conference on Cluster Computing (CLUSTER
• 作者: Evans, Richard Todd