FP4S: Fragment-based Parallel State Recovery for Stateful Stream Applications

Streaming computations are by nature long-running. They run in highly dynamic distributed environments where many stream operators may leave or fail at the same time. Most of them are stateful, in which stream operators need to store and maintain large-sized state in memory, resulting in expensive time and space costs to recover them. The state-of-the-art stream processing systems offer failure recovery mainly through three approaches: replication recovery, checkpointing recovery, and DStream-based lineage recovery, which are either slow, resource-expensive or fail to handle many simultaneous failures.We present FP4S, a novel fragment-based parallel state recovery mechanism that can handle many simultaneous failures for a large number of concurrently running stream applications. The novelty of FP4S is that we organize all the application’s operators into a distributed hash table (DHT) based consistent ring to associate each operator with a unique set of neighbors. Then we divide each operator’s in-memory state into many fragments and periodically save them in each node’s neighbors, ensuring that different sets of available fragments can reconstruct lost state in parallel. This approach makes this failure recovery mechanism extremely scalable, and allows it to tolerate many simultaneous operator failures. We apply FP4S on Apache Storm and evaluate it using large-scale real-world experiments, which demonstrate its scalability, efficiency, and fast failure recovery features. When compared to the state-of-the-art solutions (Apache Storm), FP4S reduces 37.8% latency of state recovery and saves more than half of the hardware costs. It can scale to many simultaneous failures and successfully recover the states when up to 66.6% of states fail or get lost.

流计算本质上是长期运行的。它们在高度动态的分布式环境中运行，许多流运算符可能同时离开或失败。它们中的大多数都是有状态的，流运算符需要在内存中存储和维护大容量的状态，从而导致恢复它们需要昂贵的时间和空间成本。最先进的流处理系统主要通过三种方法提供故障恢复：复制恢复、检查点恢复和基于 DStream 的行恢复，这些方法要么速度慢、资源消耗大，要么无法同时处理多个故障。FP4S 的新颖之处在于，我们将应用程序的所有操作符组织成一个基于分布式哈希表（DHT）的一致环，将每个操作符与一组唯一的邻居关联起来。然后，我们将每个操作员的内存状态划分为多个片段，并定期将其保存在每个节点的邻居中，确保不同的可用片段集可以并行重建丢失的状态。这种方法使故障恢复机制具有极高的可扩展性，可以同时容忍多个操作员故障。我们在 Apache Storm 上应用了 FP4S，并通过大规模实际实验对其进行了评估，结果证明了它的可扩展性、效率和快速故障恢复功能。与最先进的解决方案（Apache Storm）相比，FP4S 减少了 37.8% 的状态恢复延迟，节省了一半以上的硬件成本。它可以同时处理多个故障，并在多达 66.6% 的状态失效或丢失时成功恢复状态。