Apache: Big Data North America 2017

There are a few streaming platforms which provide the exactly-once processing guarantee. This is done by checkpointing the state of the functional units (operators) that make up the streaming pipeline. Many real-world big data pipelines are typically composed of operators which maintain a large ever-growing state. However, periodically checkpointing the state of these operators is only practical when their state is small. To solve this problem, I created Managed State for the Apache Apex project, which is an incrementally checkpointed key-value data structure. Additionally, the community has developed a layer ontop of Managed State (Spillable Datastructures), which allows us to incrementally checkpoint a variety of common data structures. This presentation will cover the challenges of implementing fault-tolerant incremental checkpoint in Managed State.