452 Lecture 2: RPC Admin: I am not Tom, am an impostor: Dan Ports, drkp@cs This week is about how to communicate between distributed systems. Today may be review for some. Today: RPC: Remote Procedure Call A common pattern for communication in distributed systems and how we can build some infrastructure to automate many of the details Pattern: client server request (w/ args) -------> processing waiting response <------- First point is that you’ve been doing RPC’s since you were 3. The web: click on a link, sends a message to the server, waits for a response, displays the response. Even have arguments: type search keywords, get response. File systems: client requests to read or modify a file stored on server ---------- Suppose I wanted to write a simple banking system... only on one host for now. float balance(int accountID) { return balance[accountID]; } void deposit(int accountID, float amount) { balance[accountID] += amount return OK; } client() { deposit(42, $50.00); print balance(42); } No problem calling deposit() and balance() -- these are just standard function calls Of course, we probably want our client to be able to run on a different machine, for all the usual reasons. So what would we have to do to call the balance/deposit functions on a remote machine. Define a protocol: request "balance" = 1 { arguments { int accountID (4 bytes) } response { float balance (8 bytes); } } request "deposit" = 2 { arguments { int accountID (4 bytes) float amount (8 bytes) } response { } } client() { s = socket(UDP) msg = {2, 42, 50.00} // marshalling send(s, server_address, msg) response = receive(s) check response == "OK" msg = {1, 42} send(s -> server_address, msg) response = receive(s) print "balance is" + response } server() { s = socket(UDP) bind s to port 1024 while (1) { msg, client_addr = receive(s) type = byte 0 of msg if (type == 1) { account = bytes 1-4 of msg // unmarshalling result = balance(account) send(s -> client_addr, result) } else if (type == 2) { account = bytes 1-4 of msg amount = bytes 5-12 of msg deposit(account, amount) send(s -> client_addr, "OK") } } Some details along the way: a socket is our portal for communicating with the outside world What is the server address? IP address + port, e.g. 128.208.5.65 : 1024 How do we know the IP address? Probably from DNS Why do we need the port? To identify a program on the particular machine; there might be more than one. How do we know the port number? Servers usually operate on standardized, well-known port numbers, e.g. 25 for email, 80 for HTTP OK, so we wrote a client/server application. But: - it was tedious and we are lazy - had to get the marshalling and unmarshalling exactly right on both ends; won't work if we're off by even one byte (and this is a simple example, and I waved my hands a bit; what about byte ordering, requests too large for 1 msg, etc) - this client looks nothing like the "client" we had before So you should be thinking: can we automate this process? Yes: if we wrote the protocol in a machine-readable way, we could generate a lot of this code. protocol ---RPC compiler---> client stubs, server stubs stubs are glue code that handle the communication details Client stub: deposit_stub(int account, float amount) { // marshall request type + arguments into buffer // send request to client // wait for reply // decode response // return result } To the client, looks like calling the deposit function we started with! Server stub: loop: wait for command decode and unpack request parameters call procedure build reply message containing results send reply pretty much exactly the code we just wrote for the server! To the server code (deposit, balance) looks just like the client is calling the procedures directly! Client code -> stub -> network <- stub <- server code The point of all this: transparency! - try to make remote procedure calls look just like local procedure calls - note that we could reuse the entire single-machine code we started with, and add the stubs to make it run on a distributed system. That seems great! Examples of RPC systems that do this sort of stub generation: - Sun rpcc (used in NFS paper) - XMLRPC/SOAP - Google Protocol Buffers - Java RMI How do the stubs find the server address? Could hardcode it -- inflexible Could ask a name service that maps service name to host, port ---------- Is RPC really transparent: can we really just treat remote procedure calls as ordinary procedure calls? Not quite - performance local call: maybe 10 cycles = ~3 ns RPC: 0.1-1 ms on a LAN => ~10K-100K slower in the wide area: can easily be millions of times slower - failures what happens if messages get dropped, the client or server crashes, etc? - also security, concurrent requests, etc. What kinds of failures are there? communication failures (messages may be delayed, variable round trip, never arrive) machines failures -- either client or server sometimes can't tell if it was a dropped request message or a dropped reply message or whether it was a communication failure or a machine failure or, if machine failure, whether it crashed before or after processing the request What semantics do we get with our RPC implementation above? Just hangs if there's a failure. Not good. Slightly better to timeout and tell the application we failed Also: might execute a request twice due to duplicate packet Alternative: at least once retry until we get a successful response Alternative: at most once give each request an ID and have the server keep track of whether it's been seen before dealing with server failures also a problem Which do you think is best? At-least-once versus at-most-once? let's take an example: buy a book 1.if client and server stay up, book will arrive on your kindle 2. if client fails, user may not know if book was purchased (need a plan!) 3. if server fails, client may have bought the book, or not at-least-once: client keeps trying at-most-once: client will receive an exception what does a client do in the case of an exception? Need some application-specific protocol Ex: ask server, did user buy the book? Means server needs to have a plan for remembering state across reboots at-least-once (if we never give up) clients keep trying. server may run procedure several times. server must use application state to handle duplicates if requests are not idempotent (and difficult to make all request idempotent) (for example: is reading a web page idempotent? Normally, but not always!) Ex: server logs purchases on disk, with request ID. Check log before each request, so if this is a retry, can squelch it (Actually Amazon doesn’t do this – they purchase you the second book – better to be fast and sorry than slow and correct.) e.g., server good store on disk who has lock and req id check table for each requst even if server fails and reboots, we get correct semantics What is right? depends where RPC is used. For example: file systems. Applications running on the file system don’t know whether files are local or remote. So no What then? NFS – at least once, and just deal with the occasional blurp Other network file systems: at most once, but inside the file system mask the problem from the user more sophisticated applications: need an application-level plan in both cases not clear at-once gives you a leg up => Handling machine failures makes RPC different than procedure calls ---------- Impact: RPC is used everywhere automatic marshalling is really useful; all kinds of libraries for this client stubs & transparency are useful, but transparency only goes so far dealing with failures is still hard and it typically still requires application involvement Next paper: NFS Widely used even today; basic design still looks the same Example of a system that uses RPC Runs into similar challenges plus a few more: failures, security, recovery makes a set of design decisions, that you can agree or disagree with. Not all systems do things the same way as NFS, but they face many of the same issues and they have similar options.