HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

HEC：协作研究：SAM^2 工具包：用于高端计算的可扩展和自适应元数据管理

• 批准号: 0621526
• 负责人: Hong Jiang
• 金额: --
• 依托单位: University of Nebraska-Lincoln
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-08-15 至 2010-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0621526&HistoricalAwards=false
• 关键词: HEC; Collaborative; Research; SAM; Toolkit; HEC; Collaborative; Research; SAM; Toolkit

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires ahigher level of scalability and dependability than that provided by current file and storage systems.The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to realapplication performance improvements.The project involves the following components:1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns.2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching.4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching.5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies.

高端计算应用程序对EXA-BYTE级存储容量的需求不断提高，需要比当前文件和存储系统提供的可扩展性和可靠性级别。该提案涉及HEC环境中基于可扩展集群的元数据管理的文件系统研究。它旨在开发可扩展和自适应的元数据管理（SAM2）工具包，以扩展并充分利用高性能计算社区使用的基于最新集群和分布式文件存储系统所承诺的峰值性能。关于数据流动和管理扩展的研究大量研究，但是，需要将基于群集的文件系统的属性扩展，即元数据被低估了。了解元数据流量的特征，以及适当的负载平衡，缓存，预取和分组机制相应地执行元数据管理的应用将导致高扩展性。可以预料，通过将可扩展和自适应的元数据管理组件适当地插入基于最先进的集群的并行和分布式文件存储系统中，一个人可能有可能提高应用程序和文件系统的性能，并有助于将此类系统的高峰值性能的前景和潜力转化为实用性绩效改进。项目涉及以下组件。开发多变量的预测模型来分析和预测文件元数据访问模式。2。使用重复的BLOOM滤清器阵列技术来开发可扩展和自适应文件名映射方案，以实施负载余额并提高可扩展性3。开发分散的，局部感知的元数据分组方案，以促进诸如预取的批量元数据操作。4。使用用于客户端和服务器端元数据缓存的分布式共享对象模型来开发自适应缓存相干协议。5。原型将SAM2组件纳入最新的平行虚拟文件系统PVFS2和分布式存储数据缓存系统，为Nebraska-Lincoln大学DOE CMS 2站点建立了实验框架，并进行基准测试，评估和验证研究。