Collaborative Research: Elements: Shared Data-Delivery Infrastructure to Enable Discovery with Next Generation Dark Matter and Computational Astrophysics Experiments

协作研究：要素：共享数据传输基础设施，以实现下一代暗物质和计算天体物理实验的发现

• 批准号: 2103778
• 负责人: Matthew Turk
• 金额: $ 26.4万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2103778&HistoricalAwards=false
• 关键词: Collaborative; Research; Elements; Shared; Data

Modern laboratories provide unprecedented sensitivity to the many different galactic-messengers that stream through our planet by the minute: cosmic rays, light from distant galaxies, elusive neutrinos, and possibly dark matter. Combining this information with models and data from simulations provides insight into how our universe began and continues to evolve -- the scales at which objects first collapsed, the development of stars and galaxies, and the dynamics within our own galaxy.However, this data is often inaccessible: scientists within an experiment or community struggle with the complex, custom-built programs they use to access the data. And switching to a standard format is usually not an option: these data formats are designed for requirements that often do not include cross-experiment synthesis or linking.Junior scientists - let alone the public - can struggle to generate new insights from the data because the data is difficult to access, understand and analyze. The cross-cutting inquiry that could arise from clever reuse and combination of data from different experiments and simulations is rarely conducted.This project makes data accessible both within and across collaborations, providing the infrastructure to search for signals in detectors across the globe. Extending existing efforts to improve data access makes this project possible: yt is software that provides uniform access to simulation data; Kaitai is a data-description language that enables easy access to any data format; Rucio and other tools provide a standard interface that allows data downloads; and ServiceX can identify, subset and process data with little effort from the end user.Scientists have built experiments that offer an incredible wealth of information about our world. This project works to make that information accessible to everyone.Technical DescriptionThe Personal Data-Delivery infrastructure (PONDD) addresses the data challenges of existing dark matter and astrophysics experiments while requiring no changes to existing data formats. This non-invasive, no-changes-necessary support for any file format provides opportunities to expand beyond our two identified use cases, dark matter searches and astrophysics simulations, into many other data-driven science domains that rely on custom file formats.This work delivers an infrastructure that seamlessly delivers data in a well-supported format (such as Parquet) from multiple sources. To successfully deliver cross-experiment data to end users, we bring together ongoing projects from High Energy Physics and the broader NSF community; while this project will involve development of software products (yt and Kaitai) it will also include synthesis of existing investments in cyberinfrastructure and efforts to improve their long-term sustainability.This project is supported by the Office of Advanced Infrastructure in the Directorate for Computer and Information Science and Engineering and the Division of Physics in the Directorate for Mathematical and Physical Sciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代实验室对每分钟流经我们星球的许多不同的银河信使提供了前所未有的灵敏度：宇宙射线、来自遥远星系的光、难以捉摸的中微子，以及可能的暗物质。 将这些信息与来自模拟的模型和数据相结合，可以深入了解我们的宇宙如何开始并继续演化——物体首次坍缩的尺度、恒星和星系的发展以及我们自己星系内的动力学。然而，这些数据是通常无法访问：实验或社区中的科学家们正在努力应对他们用来访问数据的复杂的定制程序。 切换到标准格式通常不是一个选择：这些数据格式是为通常不包括交叉实验合成或链接的要求而设计的。初级科学家 - 更不用说公众 - 可能很难从数据中产生新的见解，因为数据难以访问、理解和分析。 巧妙地重用和组合来自不同实验和模拟的数据可能产生的跨领域查询很少进行。该项目使数据在协作内部和跨协作之间可访问，为搜索全球探测器中的信号提供了基础设施。 扩展现有的努力以改进数据访问使该项目成为可能：yt 是提供对模拟数据的统一访问的软件； Kaitai 是一种数据描述语言，可以轻松访问任何数据格式； Rucio和其他工具提供了允许数据下载的标准接口； ServiceX 可以识别、子集化和处理数据，最终用户只需付出很少的努力。科学家们已经进行了实验，提供了有关我们世界的令人难以置信的丰富信息。 该项目致力于让每个人都能访问这些信息。技术说明个人数据传输基础设施 (PONDD) 解决了现有暗物质和天体物理实验的数据挑战，同时无需更改现有数据格式。 这种对任何文件格式的非侵入性、无需更改的支持提供了扩展到我们确定的两个用例（暗物质搜索和天体物理模拟）之外的机会，扩展到许多其他依赖自定义文件格式的数据驱动的科学领域。这项工作提供了一个基础设施，可以从多个来源以受良好支持的格式（例如 Parquet）无缝地提供数据。 为了成功向最终用户提供交叉实验数据，我们汇集了高能物理和更广泛的 NSF 社区正在进行的项目；虽然该项目将涉及软件产品（yt 和 Kaitai）的开发，但它还将包括对网络基础设施现有投资的综合以及提高其长期可持续性的努力。该项目得到了计算机和技术管理局高级基础设施办公室的支持该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。