Collaborative Research: SI2-SSI: Big Weather Web: A Common and Sustainable Big Data Infrastructure in Support of Weather Prediction Research and Education in Universities

合作研究：SI2-SSI：大天气网：支持大学天气预报研究和教育的通用且可持续的大数据基础设施

• 批准号: 1450488
• 负责人: Carl Maltzahn
• 金额: $ 69.55万
• 依托单位: University of California-Santa Cruz
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-08-01 至 2019-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1450488&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSI; Big

Earth science communities need to rely on access to large and growing amounts of curated data to make progress in research and provide adequate education. Existing infrastructures pose significant barriers to this access, especially for small to mid-size research groups and primarily undergraduate institutions: cloud services disappear when funding runs out to pay for them and therefore do not provide the long-term availability required for curated data. Similarly, in-house IT infrastructure is maintenance-intensive and requires dedicated resources for which long-term funding is often unavailable. The goal of the Big Weather Web is to make in-house IT infrastructure affordable by combining the application of three recent technologies: virtualization, federated smart storage, and big data management. Virtualization allows push-button deployment and maintenance of complex systems, smart storage provides automatic, community-wide data availability guarantees, and big data management allows for easy curation of data and its products. The combination of these three technologies allows communities to create a standard community-specific computational environment and efficiently refine it with minimal repetition of work, introducing a high degree of reproducibility in research and education. This reproducibility accelerates learning and amplifies everyone?s contribution. Due to virtualization, it can easily take advantage of cloud services whenever they become available, and it can run on in-house IT infrastructure using significantly reduced maintenance resources. The Big Weather Web will be developed in the context of numerical weather prediction with the expectation that the resulting infrastructure can be easily adapted to other data-intensive scientific communities.The volume, variety, and velocity of scientific data generated is growing exponentially. Small to mid-size research groups and especially primarily undergraduate institutions (PUIs) do not have the resources to manage large amounts of data locally and share their data products globally at high availability. This lack of resources has a number of consequences in education and research that have been well-documented in recent EarthCube workshops: (1) data-intensive scientific results are not easily reproducible, whether in the context of research or education, (2) limited or non-existent availability of intermediate results causes a lot of unnecessary duplication of work and makes learning curves unnecessarily steep, and consequently (3) scientific communities of practice are falling behind technological innovations. This Big Weather Web project focuses on the numerical weather prediction community. Numerical weather models produce terabytes of output per day, comprising a wealth of information that can be used for research and education, but this amount of data is difficult to transfer, store, or analyze for most universities. The Big Weather Web addresses this situation with the design, implementation, and deployment of "nuclei," which are shared artifacts that enable reliable and efficient access and sharing of data, encode best practices, and are sustainably maintained and improved by the community. These nuclei use existing and well-established technologies, but the integration of these technologies will significantly reduce the resource burden mentioned above. Nucleus 1 is a large ensemble distributed over seven universities. Nucleus 2 is a common storage, linking, and cataloging methodology implemented as an appliance-like Data Investigation and Sharing Environment (DISE) that is extremely easy to maintain and that automatically ensures data availability and safety. Nucleus 3 is a versioned virtualization and container technology for easy deployment and reproducibility of computational environments. Together, these nuclei will advance discovery and understanding through sharing of data products and methods to replicate scientific results while promoting teaching, training, and learning by creating a shared environment for scientific communities of practice. These shared environments are particularly important for underrepresented groups who otherwise have limited access to knowledge that is primarily propagated by social means. Our approach is a significant step towards improving reproducibility in the complex computational environments found in many scientific communities.

地球科学界需要依靠获取大量且不断增长的精选数据来取得研究进展并提供充分的教育。现有的基础设施对这种访问构成了重大障碍，特别是对于中小型研究团体和主要是本科院校来说：当资金耗尽时，云服务就会消失，因此无法提供策划数据所需的长期可用性。同样，内部 IT 基础设施是维护密集型的，需要专门的资源，而这些资源往往无法获得长期资金。 Big Weather Web 的目标是通过结合虚拟化、联合智能存储和大数据管理这三种最新技术的应用，使内部 IT 基础设施变得经济实惠。虚拟化允许复杂系统的按钮式部署和维护，智能存储提供自动的、社区范围内的数据可用性保证，大数据管理允许轻松管理数据及其产品。这三种技术的结合使社区能够创建一个标准的社区特定计算环境，并以最少的重复工作有效地完善它，从而在研究和教育中引入高度的可重复性。这种可重复性加速了学习并扩大了每个人的贡献。由于虚拟化，它可以在云服务可用时轻松利用云服务，并且可以在内部 IT 基础设施上运行，使用显着减少的维护资源。大天气网将在数值天气预报的背景下开发，期望由此产生的基础设施可以轻松适应其他数据密集型科学界。生成的科学数据的数量、种类和速度呈指数级增长。中小型研究团体，尤其是本科院校 (PUI) 没有资源在本地管理大量数据并在全球范围内以高可用性共享其数据产品。资源缺乏对教育和研究产生了一系列后果，这些后果在最近的 EarthCube 研讨会上得到了详细记录：（1）无论是在研究还是教育背景下，数据密集型科学成果都不容易重现，（2）有限中间结果的不存在或不存在会导致大量不必要的重复工作，并使学习曲线不必要地陡峭，因此（3）科学实践社区落后于技术创新。这个大天气网络项目专注于数值天气预报社区。数值天气模型每天产生数 TB 的输出，其中包含可用于研究和教育的大量信息，但对于大多数大学来说，如此大量的数据难以传输、存储或分析。大天气网通过“核心”的设计、实施和部署来解决这种情况，“核心”是共享的工件，可以可靠、高效地访问和共享数据，编码最佳实践，并由社区持续维护和改进。这些核使用现有且成熟的技术，但这些技术的集成将显着减轻上述资源负担。 Nucleus 1 是一个分布在七所大学的大型系统。 Nucleus 2 是一种通用的存储、链接和编目方法，作为类似设备的数据调查和共享环境 (DISE) 实现，该环境非常易于维护，并可自动确保数据可用性和安全性。 Nucleus 3 是一种版本化虚拟化和容器技术，可轻松部署和重现计算环境。这些核心将共同通过共享数据产品和复制科学成果的方法来促进发现和理解，同时通过为科学实践社区创建共享环境来促进教学、培训和学习。这些共享环境对于代表性不足的群体尤其重要，否则他们获得主要通过社会手段传播的知识的机会有限。我们的方法是提高许多科学界复杂计算环境中的可重复性的重要一步。