SPX: Collaborative Research: NG4S: A Next-generation Geo-distributed Scalable Stateful Stream Processing System

SPX：合作研究：NG4S：下一代地理分布式可扩展状态流处理系统

• 批准号: 1919126
• 负责人: Liting Hu
• 金额: $ 29.93万
• 依托单位: Florida International University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2022-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919126&HistoricalAwards=false
• 关键词: SPX; Collaborative; Research; NG4S; Next; SPX; Collaborative; Research; NG4S; Next

Our society increasingly relies on applications that process streaming data across geo-distributed sites, such as making business decisions from marketing data, identifying spam campaigns in social network streams, and analyzing genome datasets in different labs and countries to track the sources of potential epidemics. State-of-art solutions for these needs are centered around stateless stream processing. This project advances stream processing to enable next-generation streaming applications to store and update state along with computation, therefore processing live data streams in a timely fashion from massive and geo-distributed datasets. Existing systems are mainly designed for stateless stream processing in intra-datacenter settings and do not scale well for running stream applications that contain large distributed states. This project breaks the traditional abstractions of a centralized architecture and hashtable-based stateless operators, redefining them with a new decentralized architecture and new memory-efficient stateful operators, which enables novel approaches to improve overall system performance and scalability. This project builds a next-generation geo-distributed scalable stateful stream processing system that will significantly improve the scalability of stream processing systems. This work includes three primary research directions. (1) At the architecture level, a new decentralized 'many masters/many workers' architecture will be proposed, which provides each master with maximum independence. (2) At the operator level, a new in-memory data structure will be designed and implemented to store application state and minimize the memory overhead so as to handle 'big data' requirements. (3) A new shard-based parallel recovery mechanism will be proposed to handle failures and stragglers in a scalable way. All three parts of the project will be prototyped and implemented on a widely adopted stream processing system (Apache Storm).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们的社会越来越依赖于处理跨地理分布站点的流数数据的应用程序，例如从营销数据中做出业务决策，识别社交网络流中的垃圾邮件活动，并分析不同实验室和国家 /地区的基因组数据集以跟踪潜在流行病的来源。满足这些需求的最新解决方案围绕无状态流处理。该项目推进了流处理，以启用下一代流媒体应用程序与计算一起存储和更新状态，因此从大型和地理分布的数据集中及时处理实时数据流。现有系统主要是为无状态的流化处理设置中的无状态流处理而设计的，对于运行包含大型分布式状态的流式应用程序的扩展不佳。该项目打破了集中式体系结构和基于Hashtable的无状态运营商的传统抽象，通过新的分散体系结构和新的记忆有效的状态运营商重新定义了它们，这使得可以提高整体系统性能和可扩展性。 该项目构建了下一代地理分配的可扩展状态流处理系统，该系统将显着提高流处理系统的可扩展性。这项工作包括三个主要研究方向。 （1）在体系结构层面上，将提出一个新的分散的“许多大师/许多工人的体系结构，为每个主人提供最大的独立性。 （2）在操作员级别上，将设计和实施一种新的内存数据结构，以存储应用程序状态并最大程度地减少内存开销，以便处理“大数据”要求。 （3）将提出一种新的基于碎片的平行恢复机制，以可扩展的方式处理故障和散落者。该项目的所有三个部分将在广泛采用的流处理系统（Apache Storm）上进行原型化和实施。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准，认为值得通过评估来获得支持。