CSE 599c Hot Topics in Data Management Systems

Lecture 9: Distributed Stream Processing

Fault-Tolerance

Magdalena Balazinska

Winter 2006

Today's Papers

J.-H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, M. Stonebraker, and S. Zdonik. High-Availability Algorithms for Distributed Stream Processing. ICDE 2005.

M. Balazinska, H. Balakrishnan, S. Madden, and M. Stonebraker. Fault-Tolerance in the Borealis Distributed Stream Processing System. SIGMOD 2005.

Fault-Tolerance Problem

Node 2

Borealis Node 1

Node 3

...

Data

Source

Data

Source

...

Join

Union

Agg.

Agg.

Map

Filter

Node Failures

Network Failures and Partitions

Possible types of failures?

Consequences of these failures?

Types of Failures

Processing nodes can fail

Crash failures

Byzantine failures -> not discussed

Network can fail

Some nodes may be unable to communicate

The system can partition

Data source failures -> not discussed

Consequences of Failures

Loss of state

Inaccurate results

Blocked operators

Outline

Node failures

How to recover the state of a failed node?

Performance vs overhead of different techniques

Operator types and recovery guarantees

Network failures

How to handle network failures and partitions?

Approach based on tentative and stable tuples

Availability/consistency trade-offs

Handling complex failure scenarios

Recovering State of Failed Node

Basic approach

Each primary processing node has a secondary

When primary fails, secondary takes over

Possible techniques

Passive Standby

Active Standby

Upstream Backup

N4

Primary N3

Passive Standby

...

N1

...

...

...

N2

N5

N6

...

Secondary N3'

...

...

...

...

How to prevent losses?

How to avoid duplications?

periodic

checkpoints

Passive Standby Guarantees

Basic variant provides rollback recovery

Deterministic query diagrams: repeating

Non-determinstic query diagrams: divergent

To achieve precise recovery

Deterministic: trivial duplicate elimination

Non-deterministic: checkpoint outputs tuples before sending them downstream

Passive Standby Performance

Recovery time dominated by

reprocessing on average half checkpoint interval

Runtime overhead

grows with size of checkpoint messages

decreases as checkpoint interval increases

Checkpoint interval determines trade-off between recovery time and runtime overhead

Active Standby

N4

Primary N3

...

N1

...

...

...

N2

N5

N6

...

Secondary N3'

...

...

...

...

When to drop output tuples at secondary?

How to avoid duplications?

Secondary processes in parallel

Active Standby Guarantees

Basic variant provides rollback recovery

Deterministic query diagrams: repeating

Non-determinstic query diagrams: divergent

To achieve precise recovery

Deterministic: trivial duplicate elimination

Non-deterministic: must send determinants from primary to secondary to ensure they follow the same execution

Active Standby Performance

Recovery time

negligible, only need to resend duplicate tuples

Runtime overhead

over 100%, as all processing is duplicated

Output trimming interval determines trade-off between recovery time and runtime overhead

Upstream Backup

N4

Primary N3

...

N1

...

...

...

N2

N5

N6

...

Secondary N3'

...

...

...

...

What to keep upstream?

How to avoid duplications?

Secondary does nothing

Ack 0

cause

Ack 1

Upstream Backup

Upstream Backup

Upstream Backup Guarantees

Basic variant provides rollback recovery

Repeatable query diagrams: repeating

Convergent-capable: convergent

Other query diagrams: divergent

To achieve precise recovery

Repeatable: trivial duplicate elimination

Convergent-capable: use high watermarks

Other query diagrams: use determinants

Upstream Backup Performance

Recovery time dominated by

rebuilding the whole state from logs

Runtime overhead

bw overhead negligible

Ack interval determines trade-off between recovery time and runtime overhead

Recovering Failed Nodes Summary

Network Failures and Partitions

Availability: Low per-tuple processing latency

Consistency: Output identical to failure-free execution

Data

Source

Union

...

Agg.

Node 1

Cannot achieve both availability and consistency when network partitions

Data

Source

Application Requirement

Network monitoring

Useful to hear even from subset of monitors

Need to know what happened during a failure

Military applications

Current location of any enemy unit

Earlier movements of enemy units

Financial services

Continue even if some market feeds fail

Eventually need complete, correct analysis

Fault-Tolerance Goals

Minimize inconsistency subject to incremental_delay < X

Goal 1: Maintain required availability

Incremental latency within pre-defined bound

Goal 2: Eventual consistency

Correct results after failure heals

Goal 3: Minimize inconsistency

# tuples generated with missing inputs

Example Output Seen By Clients

Suspend

24

23

Tentative

25

UNDO

22

Reconciliation

Stable

23

24

25

Failure

22

Stable

Stable state

21

31

Stable state

32

REC_DONE

30

27

28

Clearing queues

29

30

26

22

21

24

23

26

25

28

27

30

29

32

31

Execution without failure

Execution with failure

Protocol State Machine

STABLE

UPSTREAM

FAILURE

Missing or

TENTATIVE inputs

Failure heals

Another upstream failure in progress

Reconcile state

Correct output

STABILIZATION

Maintain replica consistency Detect failures of inputs

Ensure low proc. latency Reduce inconsistency

Ensure eventual consistency Maintain low proc. latency

Protocol Properties

Does the approach achieve the required processing latency for multi-node networks?

How much consistency can we gain by delaying tentative tuples?

SO

SUnion

Node 1

I2

I3

I1

SO

SUnion

Node 2

SO

SUnion

Node 3

SO

SUnion

Node 4

SO

SUnion

Node 1'

SO

SUnion

Node 2'

SO

SUnion

Node 3'

SO

SUnion

Node 4'

Delay & Delay

Process & Process

Exploit Flexible Availability Req.

Delaying Improves Consistency

5-second failure

60-second failure

No Improvement with
Long-Duration Failures

Conclusion

Many types of failures

Node failures, network failures, etc.

Trade-offs in

Fault-tolerance guarantees

Recovery performance

Runtime overhead

Results independent of failure duration!

So scheme does work.

So there's no point delaying

Block everyone at thebeginning.

2006-01-01T23:16:26