Collaborative Research: SI2-SSI: Big Weather Web: A Common and Sustainable Big Data Infrastructure in Support of Weather Prediction Research and Education in Universities

合作研究：SI2-SSI：大天气网：支持大学天气预报研究和教育的通用且可持续的大数据基础设施

• 批准号: 1450439
• 负责人: Allen Evans
• 金额: $ 16.44万
• 依托单位: University of Wisconsin-Milwaukee
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-08-01 至 2019-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1450439&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSI; Big

Earth science communities need to rely on access to large and growing amounts of curated data to make progress in research and provide adequate education. Existing infrastructures pose significant barriers to this access, especially for small to mid-size research groups and primarily undergraduate institutions: cloud services disappear when funding runs out to pay for them and therefore do not provide the long-term availability required for curated data. Similarly, in-house IT infrastructure is maintenance-intensive and requires dedicated resources for which long-term funding is often unavailable. The goal of the Big Weather Web is to make in-house IT infrastructure affordable by combining the application of three recent technologies: virtualization, federated smart storage, and big data management. Virtualization allows push-button deployment and maintenance of complex systems, smart storage provides automatic, community-wide data availability guarantees, and big data management allows for easy curation of data and its products. The combination of these three technologies allows communities to create a standard community-specific computational environment and efficiently refine it with minimal repetition of work, introducing a high degree of reproducibility in research and education. This reproducibility accelerates learning and amplifies everyone?s contribution. Due to virtualization, it can easily take advantage of cloud services whenever they become available, and it can run on in-house IT infrastructure using significantly reduced maintenance resources. The Big Weather Web will be developed in the context of numerical weather prediction with the expectation that the resulting infrastructure can be easily adapted to other data-intensive scientific communities.The volume, variety, and velocity of scientific data generated is growing exponentially. Small to mid-size research groups and especially primarily undergraduate institutions (PUIs) do not have the resources to manage large amounts of data locally and share their data products globally at high availability. This lack of resources has a number of consequences in education and research that have been well-documented in recent EarthCube workshops: (1) data-intensive scientific results are not easily reproducible, whether in the context of research or education, (2) limited or non-existent availability of intermediate results causes a lot of unnecessary duplication of work and makes learning curves unnecessarily steep, and consequently (3) scientific communities of practice are falling behind technological innovations. This Big Weather Web project focuses on the numerical weather prediction community. Numerical weather models produce terabytes of output per day, comprising a wealth of information that can be used for research and education, but this amount of data is difficult to transfer, store, or analyze for most universities. The Big Weather Web addresses this situation with the design, implementation, and deployment of "nuclei," which are shared artifacts that enable reliable and efficient access and sharing of data, encode best practices, and are sustainably maintained and improved by the community. These nuclei use existing and well-established technologies, but the integration of these technologies will significantly reduce the resource burden mentioned above. Nucleus 1 is a large ensemble distributed over seven universities. Nucleus 2 is a common storage, linking, and cataloging methodology implemented as an appliance-like Data Investigation and Sharing Environment (DISE) that is extremely easy to maintain and that automatically ensures data availability and safety. Nucleus 3 is a versioned virtualization and container technology for easy deployment and reproducibility of computational environments. Together, these nuclei will advance discovery and understanding through sharing of data products and methods to replicate scientific results while promoting teaching, training, and learning by creating a shared environment for scientific communities of practice. These shared environments are particularly important for underrepresented groups who otherwise have limited access to knowledge that is primarily propagated by social means. Our approach is a significant step towards improving reproducibility in the complex computational environments found in many scientific communities.

地球科学社区需要依靠获取大量和越来越多的策划数据来取得研究进展并提供足够的教育。现有的基础设施对此访问构成了重大障碍，尤其是对于中小型研究小组，主要是本科机构：云服务在资金用尽以支付这些费用时消失，因此不提供策划数据所需的长期可用性。同样，内部的IT基础设施是维护的，需要专门的资源，而这些资源通常不可用。大天气网的目的是通过结合三种最近的技术的应用来使内部IT IT基础架构负担得起：虚拟化，联合智能存储和大数据管理。虚拟化允许按钮的部署和维护复杂系统，Smart Storage提供自动，社区范围内的数据可用性保证，而大数据管理则可以轻松策划数据及其产品。这三种技术的结合使社区可以创建一个标准的社区特定计算环境，并通过最少的工作重复来有效地完善它，从而在研究和教育中引入了高度的可重复性。这种可重复性可以加速学习并扩大每个人的贡献。由于虚拟化，每当云服务可用时，它都可以轻松利用云服务，并且可以使用大幅减少的维护资源在内部IT IT基础架构上运行。大天气网络将在数值天气预测的背景下开发，并期望由此产生的基础设施很容易适应其他数据密集型科学群落。所产生的科学数据的数量，多样性和速度正在呈指数增长。中小型研究小组，尤其是主要是本科机构（PUI）没有资源来管理大量数据，并以高可用性在全球范围内共享其数据产品。这种缺乏资源在最近的Earthubube研讨会中已经有充分记录的教育和研究后果：（1）数据密集型科学结果不容易重现，（（2）有限或非存在的中等结果的可用性会导致许多不必要的练习，并导致了许​​多不必要的练习，并导致了经过艰巨的态度，并且陷入了艰巨的态度，并且有必要的局限性（3岁）是不必要的，并且是不必要的（3岁），并且是不必要的，并且是不必要的，并且是不必要的，并且是不必要的，并且是不必要的效果。创新。这个大天气网络项目着重于数值天气预测界。数值天气模型每天生产出产出的trabyt，其中包括可用于研究和教育的大量信息，但是对于大多数大学来说，这一数量的数据很难转移，存储或分析。大天气网络通过“ Nuclei”的设计，实施和部署来解决这种情况，它们是共享的人工制品，可以使数据可靠，有效地访问和共享，编码最佳实践，并得到社区可持续维护和改进。这些核使用现有和公认的技术，但是这些技术的整合将大大减轻上述资源负担。核1是在七所大学上分布的大型合奏。 Nucleus 2是一种常见的存储，链接和分类方法，该方法是一种类似设备的数据调查和共享环境（DISE）实施的，非常易于维护，并自动确保数据的可用性和安全性。 Nucleus 3是一种版本的虚拟化和容器技术，可在计算环境中易于部署和可重复性。这些核共同通过共享数据产品和方法来提高发现和理解，以复制科学结果，同时通过为科学的实践社区创造共同的环境来促进教学，培训和学习。这些共享环境对于代表性不足的群体尤其重要，这些群体否则对知识的访问有限，而知识主要通过社会手段传播。我们的方法是在许多科学社区发现的复杂计算环境中改善可重复性的重要一步。