From: David V. Winkler (dwinkler_at_windows.microsoft.com)
Date: Wed Feb 25 2004 - 09:30:29 PST
Review: Cluster-Based Scalable Network Services
The paper discusses a cluster based approach to providing web services. With these services the author notes that it is more important to have availability than full consistency. This is basically the difference between ACID (atomicity, consistency, isolation, durability), and BASE (basically available, soft state, eventual consistency). This paper makes the compelling argument that commodity hardware (off the shelf PCs with fast network hardware) is the ideal building block for these clusters.
One really cool part of the paper was the overflow discussion. For periods of unusually high use additional machines may be temporarily brought into the cluster. The authors suggest the desktop machines of the company, but Figure 6 (requests per second), seem to suggest that the unusually high use will likely be during office hours, but "Often, it is during such bursts that uninterrupted operation is most critical." [a google search shows that this exact sentance is used in two distinct Fox papers]
The basic framework for the backend system supports a series of unix style pipes. Pretty extensible, but it also seems that it might have been only a slight generalization of their TranSend application, which gives lower bitrate recompressions of images for transmission over a modem. HotBot was slightly more complicated since the nodes aren't fully interchangable. The story of physically moving the cluster emphasizes the importance of availability over accuracy.
I really like figures 5 and 6. Figure 5 is a distribution of file sizes for 3 different content types, gif, jpeg, and html. The paper's analysis of this graph gives some insight into the type of data available on the web. Figure 6 shows the 24 hour period cycle of dialup showing the dialup usage with morning and late evening showing up very clearly.
This archive was generated by hypermail 2.1.6 : Wed Feb 25 2004 - 09:30:35 PST