SPX: Collaborative Research: NG4S: A Next-generation Geo-distributed Scalable Stateful Stream Processing System

SPX：合作研究：NG4S：下一代地理分布式可扩展状态流处理系统

• 批准号: 2202859
• 负责人: Liting Hu
• 金额: $ 29.93万
• 依托单位: Virginia Polytechnic Institute and State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-02-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2202859&HistoricalAwards=false
• 关键词: SPX; Collaborative; Research; NG4S; Next

Our society increasingly relies on applications that process streaming data across geo-distributed sites, such as making business decisions from marketing data, identifying spam campaigns in social network streams, and analyzing genome datasets in different labs and countries to track the sources of potential epidemics. State-of-art solutions for these needs are centered around stateless stream processing. This project advances stream processing to enable next-generation streaming applications to store and update state along with computation, therefore processing live data streams in a timely fashion from massive and geo-distributed datasets. Existing systems are mainly designed for stateless stream processing in intra-datacenter settings and do not scale well for running stream applications that contain large distributed states. This project breaks the traditional abstractions of a centralized architecture and hashtable-based stateless operators, redefining them with a new decentralized architecture and new memory-efficient stateful operators, which enables novel approaches to improve overall system performance and scalability. This project builds a next-generation geo-distributed scalable stateful stream processing system that will significantly improve the scalability of stream processing systems. This work includes three primary research directions. (1) At the architecture level, a new decentralized 'many masters/many workers' architecture will be proposed, which provides each master with maximum independence. (2) At the operator level, a new in-memory data structure will be designed and implemented to store application state and minimize the memory overhead so as to handle 'big data' requirements. (3) A new shard-based parallel recovery mechanism will be proposed to handle failures and stragglers in a scalable way. All three parts of the project will be prototyped and implemented on a widely adopted stream processing system (Apache Storm).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们的社会越来越依赖于处理跨地理分布式站点的流数据的应用程序，例如根据营销数据做出业务决策，识别社交网络流中的垃圾邮件活动，以及分析不同实验室和国家的基因组数据集以跟踪潜在流行病的来源。满足这些需求的最先进的解决方案以无状态流处理为中心。该项目推进了流处理，使下一代流应用程序能够在计算的同时存储和更新状态，从而及时处理来自海量地理分布式数据集的实时数据流。现有系统主要是为数据中心内设置中的无状态流处理而设计的，并且不能很好地扩展以运行包含大型分布式状态的流应用程序。该项目打破了集中式架构和基于哈希表的无状态运算符的传统抽象，用新的去中心化架构和新的内存高效有状态运算符重新定义了它们，从而实现了提高整体系统性能和可扩展性的新方法。 该项目构建了下一代地理分布式可扩展有状态流处理系统，将显着提高流处理系统的可扩展性。这项工作包括三个主要研究方向。 （1）在架构层面，将提出一种新的去中心化的“多master/多worker”架构，为每个master提供最大的独立性。 (2) 在操作层面，将设计并实现一种新的内存数据结构来存储应用程序状态并最小化内存开销，从而满足“大数据”需求。 （3）将提出一种新的基于分片的并行恢复机制，以可扩展的方式处理故障和掉队。该项目的所有三个部分都将在广泛采用的流处理系统（Apache Storm）上进行原型设计和实施。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

FP4S: Fragment-based Parallel State Recovery for Stateful Stream Applications

• DOI: 10.1109/ipdps47924.2020.00116
• 发表时间: 2020-05
• 期刊: 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
• 作者: Pinchao Liu;Hailu Xu;D. D. Silva-D.;Qingyang Wang;Sarker Tanzir Ahmed;Liting Hu

Achieving Online and Scalable Information Integrity by Harnessing Social Spam Correlations

• DOI: 10.1109/access.2023.3236604
• 发表时间: 2023
• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Hailu Xu;Pinchao Liu;Boyuan Guan;Qingyang Wang;Dilma Da Silva;Liting Hu

XUnified: A Framework for Guiding Optimal Use of GPU Unified Memory

XUnified：指导 GPU 统一内存优化使用的框架

• DOI: 10.1109/access.2022.3196008
• 发表时间: 2022
• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Xu, Hailu;Lin, Pei-Hung;Emani, Murali;Hu, Liting;Liao, Chunhua
• 通讯作者: Liao, Chunhua

DART: A Scalable and Adaptive Edge Stream Processing Engine

DART：可扩展的自适应边缘流处理引擎

• 发表时间: 2021
• 期刊: 2021 USENIX Annual Technical Conference (USENIX ATC 21
• 作者: Liu, Pinchao;Silva, Dilma Da;Hu, Liting.
• 通讯作者: Hu, Liting.

Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems

• DOI: 10.1109/tpds.2023.3251997
• 发表时间: 2023-08
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Hailu Xu;Pinchao Liu;Sarker Tanzir Ahmed;Dilma Da Silva;Liting Hu