Enabling data storage for petabyte-level Numerical Weather Prediction

为 PB 级数值天气预报启用数据存储

基本信息

批准号：
2533102
负责人：
金额：
--
依托单位：
University of Edinburgh
依托单位国家：
英国
项目类别：
Studentship
财政年份：
2021
资助国家：
英国
起止时间：
2021 至无数据
项目状态：
未结题

来源：
https://gtr.ukri.org/projects?ref=studentship-2533102
关键词：
Enabling data storage petabyte level

Enabling data storage petabyte level

项目摘要

Networked file systems and magnetic spinning media are nowadays a common form of hot-layer storage in HPC/HPDA platforms. New developments over the past decade in non-volatile memory, including 3DXpoint solid-state drives and storage class memory devices, have been shown to be advantageous in data intensive applications for the new trade-off they offer in latency, capacity, throughput and cost.Alongside these hardware developments, high-performance object stores have emerged offering an alternative paradigm to traditional POSIX file systems and their often hindering state management and consistency requirements. Some of these object stores have not only been designed to leverage the newer and more efficient storage media and their byte-grain random access capability, but also have been augmented by user-space IO frameworks and persistent memory programming libraries, providing a rich software landscape for implementing high-performance data storage functionality in end-user applications.This research thesis will assess the impact and suitability of these developments in the data-intensive field of numerical weather prediction, particularly by adapting ECMWF's domain-specific high-performance IO storage stack to use novel object storage technologies such as Intel DAOS, Seagate CORTX, DDN WOS, and hardware such as 3DXpoint memory. This work will be conducted in collaboration with EPCC at the University of Edinburgh.

如今，网络文件系统和磁性纺丝介质是HPC/HPDA平台中热层存储的一种常见形式。 New developments over the past decade in non-volatile memory, including 3DXpoint solid-state drives and storage class memory devices, have been shown to be advantageous in data intensive applications for the new trade-off they offer in latency, capacity, throughput and cost.Alongside these hardware developments, high-performance object stores have emerged offering an alternative paradigm to traditional POSIX file systems and their often hindering state management and consistency requirements.这些对象存储中的一些不仅旨在利用更新的，更有效的存储媒体及其字节元的随机访问能力，而且还通过用户空间IO框架和持续的内存编程库增强，还可以提供丰富的软件格局，提供丰富的软件格局，以实现在最终的研究中实现高表现数据的范围。预测，特别是通过适应ECMWF的特定领域高性能IO存储堆栈，以使用新颖的对象存储技术，例如Intel Daos，Seagate Cortx，DDN WOS和硬件，例如3DXPoint Memory。这项工作将与爱丁堡大学的EPCC合作进行。