CC* Data Storage: Institutional Storage for the University of Notre Dame (NDStore)

CC* 数据存储：圣母大学机构存储 (NDStore)

• 批准号: 2232803
• 负责人: Jaroslaw Nabrzyski
• 金额: $ 50万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2232803&HistoricalAwards=false
• 关键词: CC; Data; Storage; Institutional; University

This project equips the Center for Research Computing at the University of Notre Dame (ND CRC) and its scientific users across all Notre Dame colleges and departments to enable transformative research in social and physical sciences and engineering domains through the acquisition of institutional storage called NDStore. Major beneficiaries of NDStore are researchers utilizing various research cores at Notre Dame, such as the Genomics and Bioinformatics Core and the Notre Dame Integrated Imaging Facility, as well as other researchers from University Centers and Institutes, such as the Institute for Data and Society, in addition to the broader national community via the Open Science Grid. Together, these different facilities and researchers generate hundreds of terabytes of data per year, and they enable expert users to address the most complex research problems of today’s world. The major capabilities provided by NDStore accelerate existing research otherwise throttled by insufficient storage capability. They also enable full data lifecycle at previously inaccessible scales, enable new national data-intensive collaborations, and incubate new research projects. NDStore brings to Notre Dame an additional 2 petabytes of storage capacity for data manipulation, curation, and long-term preservation, as well as 250 terabytes of fast scratch storage for machine learning-related workloads. NDStore is a highly available solution based on an open-source Ceph-based storage clustering standard. It was designed with the flexibility to meet the needs of researchers at various stages of their research. NDStore provides a clear benefit to researchers who generate data with various instruments in core facilities and need to transfer the data to their home directories for analysis and curation before the data is shared with their communities. Before this project was funded, the amount of storage provided by ND CRC to each faculty lab was not satisfactory for most of the users dealing with large data coming from instruments at core facilities, such as microscopes, sequencing machines, or other benchtop devices. In addition, ND CRC’s high-performance scratch storage system has been shared between high-performance computing and machine learning workloads; very often, mixing these workloads led to performance bottlenecks, negatively impacting all of the storage system users at Notre Dame. NDStore helps Notre Dame create an independent scratch system for machine learning workloads.Another important aspect of the intellectual merit of this project is the opportunity for the CRC to deploy NDStore in such a way that the entire data lifecycle at Notre Dame’s research enterprise is supported. Research data use cases at ND are highly diverse, complex, and heterogeneous. They differ in types of data captured, scientific instruments used, data processing and analyses conducted, policies and methods for data sharing and use, and, internal to the lab, cyberinfrastructure-related knowledge. Data life cycle stages include: 1) data capture; 2) initial processing near the instrument(s); 3) central processing at data centers or clouds; 4) data storage, curation, and archiving; and 5) data access, dissemination, and visualization. Until NDStore was deployed, Notre Dame infrastructure could adequately support only stages 1-3 and 5, with very minimal support for stage 4. NDStore fills this gap. NDStore will also be integrated into classroom and undergraduate internship programs hosted by numerous faculty in colleges and ND CRC. Through user training, research experience for undergraduates, pre-college programs for high school students, workshops, internships, and experiential training programs, ND CRC will ensure that NDStore has the broadest possible impact on the local and national academic community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目为圣母大学研究计算中心 (ND CRC) 及其所有圣母大学学院和院系的科学用户提供装备，通过收购名为 NDStore 的机构存储，实现社会科学、物理科学和工程领域的变革性研究。 NDStore 的主要受益者是利用圣母大学各种研究核心的研究人员，例如基因组学和生物信息学核心以及圣母大学综合成像设施，以及来自大学中心的其他研究人员数据与社会研究所等研究所以及更广泛的国家社区通过开放科学网格，这些不同的设施和研究人员每年生成数百 TB 的数据，它们使专家用户能够解决大多数问题。 NDStore 提供的主要功能加速了现有的研究，否则会因存储能力不足而受到限制。它们还能够以以前无法访问的规模实现完整的数据生命周期，实现新的国家数据密集型合作，并孵化新的研究项目。 NDStore 为 Notre Dame 带来了额外的 2 PB 存储容量，用于数据操作、管理和长期保存，以及用于机器学习相关工作负载的 250 TB 快速暂存存储 NDStore 是一种基于开放式的高可用性解决方案。 -source 基于 Ceph 的存储集群标准，其设计灵活，可以满足研究人员在不同研究阶段的需求，为使用各种仪器生成数据的研究人员提供了明显的好处。在该项目获得资助之前，ND CRC 向每个教职员工实验室提供的存储量对于大多数人来说并不令人满意。处理来自核心设施仪器（例如显微镜、测序机或其他台式设备）的大数据的用户此外，ND CRC 的高性能暂存存储系统经常在高性能计算和机器学习工作负载之间共享； ，将这些工作负载混合到性能瓶颈，对 Notre Dame 的所有存储系统用户产生负面影响 NDStore 帮助 Notre Dame 创建一个用于机器学习工作负载的独立暂存系统。该项目的另一个重要方面是 CRC 有机会在此类中部署 NDStore。 ND 的研究数据用例高度多样化、复杂且异构，它们在捕获的数据类型、使用的科学仪器和数据处理方面有所不同。进行的分析、数据共享和使用的政策和方法，以及实验室内部的网络基础设施相关知识，包括：1) 数据捕获；2) 仪器附近的初始处理；在数据中心或云中进行处理；4) 数据存储、管理和归档；以及 5) 数据访问、传播和可视化，在部署 NDStore 之前，Notre Dame 基础设施只能充分支持第 1-3 和第 5 阶段。对第 4 阶段的支持非常少。NDStore 还将通过用户培训、本科生研究经验、高中预科课程纳入由大学和 ND CRC 的众多教师主办的课堂和本科生实习项目。学生、研讨会、实习和体验式培训项目，ND CRC 将确保 NDStore 对当地和国家学术界产生尽可能广泛的影响。该奖项反映了 NSF 的法定使命，并被认为值得支持通过使用基金会的智力优点和更广泛的影响审查标准进行评估。