Lecture 2: More RPC — Notes

These are notes from the lecture on April 1, 2026. See also the whiteboard descriptions and the whiteboard PDF.

These materials were drafted by AI based on the live whiteboard PDF and audio transcript from the corresponding lecture and then reviewed and edited by course staff. They may contain errors. Please let us know if you spot any.

Recap

Last time we asked what a distributed system is, why you'd want to build one, and how hard it is. We ended up with the standard fault model: the set of failures we plan to tolerate automatically. Today we try to actually build something in that fault model.

Protocols

A protocol is a description of what code and data to run and store on each machine. It is essentially a distributed algorithm. A protocol:

• Solves a particular problem
• In a particular fault model

Just like algorithms have assumptions about their input, protocols have assumptions about their environment — in particular, what kinds of failures are possible.

Correctness

A protocol is correct if on any execution where all the failures are listed in the fault model, the protocol does the right thing. We don't care if it breaks on failures outside the fault model.

Key-Value Store

In single-node programming, we have a hash map. It has get(key), put(key, value), and for our purposes also append(key, new_stuff) (which appends to whatever is currently stored at the key, treating a missing key as the empty string). Our keys and values are strings.

A key-value store is just a fancy distributed word for a hash map that's available over the network. Instead of calling a Java method on the hash map, you send it a message. This is the problem we'll solve all quarter — all four labs involve making distributed key-value stores in more and more interesting ways.

Client-server architecture

The client lives on one machine. The key-value store (the server) lives on another. The client sends requests (e.g., get("foo")) over the network. The server looks up the answer in its local data structure and sends back a response (e.g., "bar").

The interesting part is not what's inside the server — it's getting the networking right.

Serialization

When you send a message over the network, the content is bytes. Serialization converts data into bytes; deserialization converts bytes back into data. We'll mostly ignore serialization this quarter (the labs have a framework that handles it), but it's worth knowing it exists.

One important consequence: in local Java method calls, arguments are passed by reference. That doesn't work over the network — a reference to a string on your machine is meaningless on another machine (best case: segfault; worst case: points to something completely wrong). So in RPC, everything is passed by value: the entire argument is included in the message.

Remote Procedure Call (RPC)

A remote procedure call is exactly what it sounds like: calling a function on a different machine. ("Procedure" is a 1970s word for function.)

• Request message: contains the method name and all arguments
• Response message: contains success/failure and (on success) the return value

If the network works perfectly, this just works. The hard part is handling failures.

Naive RPC

Naive RPC is the simple version: client sends a request, server executes it, server sends a response. It solves the problem of calling methods remotely in the fault model where no failures are allowed.

Making RPC Work in the Standard Fault Model

The standard fault model has five components: message drops, message delay/reordering, message duplication, and machine crashes. Naive RPC doesn't handle any of them. Let's take them in order.

Dropped messages

If the request is dropped, the server never sees it. The client waits forever for a response that will never come.

If the response is dropped, the server actually executed the request, but the client never hears about it. The client also waits forever.

The key insight: from the client's perspective, these two scenarios are indistinguishable. In one case the operation never happened; in the other it did. For read requests this is annoying; for write requests it's genuinely dangerous — you don't know whether your data changed.

Retransmission

The standard fix for dropped messages: set a timer and retransmit if no response arrives.

This fixes the first scenario (dropped request). But in the second scenario (dropped response), the server already executed the request. Now the retransmitted request arrives and the server executes it again. The operation ran twice.

From the client's perspective, both scenarios look identical: send a message, hear nothing, retransmit, get a response. But the outcomes differ — one execution vs. two. So fixing the double-execution problem must happen on the server, not the client.

Idempotence (tangent)

One approach: design your API so that running a method once is equivalent to running it twice. This property is called idempotence. If your operations are idempotent, naive RPC with retransmission already works. This is why people in distributed systems advocate making REST APIs idempotent — it simplifies the system.

The problem is that not all operations are inherently idempotent (e.g., append). We need a general solution.

Naive RPC + Sequence Numbers

Attach a unique identifier — a sequence number — to each request. The client starts at some number and increments for each new request (not for retransmissions — retransmissions reuse the same sequence number).

The server stores the set of executed sequence numbers. When a request arrives, the server checks the set:

• Not seen before → execute the request, add to the set, send the response
• Seen before → it's a duplicate; don't re-execute

Multiple clients

If two clients both generate sequence numbers, their numbers can overlap. Fix: give each client a unique ID. The unique identifier for a message becomes the pair (client_id, sequence_number). The server stores these pairs rather than just sequence numbers.

In the labs, all nodes have fixed IDs assigned in advance. In the real world, you might have a separate system that assigns client IDs, or use IP addresses (with caveats).

Space optimization

The set of executed pairs grows without bound — linear in the number of requests ever sent. Optimization: instead of storing every pair, store just the highest sequence number per client. If a request arrives with a sequence number ≤ the stored highest, it's a duplicate.

This optimization requires that clients do not send concurrent requests — each client must wait for a response before sending its next request. Otherwise, reordering could cause the server to record a high sequence number before a lower-numbered request arrives, incorrectly marking it as a duplicate.

Retransmitting stored responses

There's a subtle issue: when the server detects a duplicate request, it can't just ignore it. The client retransmitted because it never got the response. If the server silently drops the duplicate, the client is stuck waiting forever — the same problem we started with.

The server must retransmit the response for duplicate requests.

This means storing not just (client_id, sequence_number) pairs, but (client_id, sequence_number, response) triples. On a duplicate request, the server looks up the stored response and sends it again without re-executing the operation.

Summary

This protocol — naive RPC with sequence numbers, duplicate detection, and response retransmission — works in the standard fault model minus machine crashes. If the server crashes, it loses its in-memory set of executed requests. Handling machine crashes is the subject of labs 2 and 3.

Lab 1 is to work out the details of this protocol and implement it.