From: Song Xue (speedsx_at_hotmail.com)
Date: Sat Jan 31 2004 - 17:44:15 PST
The paper "Experience with Grapevine: The Growth of a Distributed System" gives an structure overview of Grapevine distributed system, followed by experience gathered over the years of operations.
Grapevine is a distributed, replicated system that provides message delivery, naming, authentication, resource location, and access control services in an internet of computers. The services are divided into message service and the registration service. The message service accepts messages prepared by clients for delivery to individual recipients and distribution lists. The registration service provides naming, authentication, access control and resource location functions to clients. There are two types of entries in the registration databases: group entries and individual entries. The registration database is distributed and replicated. Associated with each registry is a group of human administrators, called registrars, who are responsible for creating, updating, and deleting individuals and groups in the registry. A client program of Grapevine generally obtains services through code, called the GrapevineUser package, that runs in the client's computer.
An objective of Grapevine design was the ability to increase system capacity over a large range by adding more servers of fixed power, rather than by using more powerful servers. Division of the registration database into registries is the primary method for preventing scaling problems. Scalability related to distribution lists across organization/geography/job/project boundaries can be better addressed by adding an indirection in their interpretation. The underlying size of the internet presents another scaling effect where the probability of direct connection decreases and messages have to travel over a series of links and gateways.
Running Grapevine requires the ability to make configuration decisions about how many servers to have, where to place them, and how to distribute registry replicas and inboxes among them. Within a local area a server will be added when the load on the existing server gets too large. In addition, when a population of users develops that is separated from the nearest server by one or more slow or unreliable network links, it usually is appropriate to add a local server. Primary inboxes are usually assigned to the server that is closest to the workstation from which a user normally reads new messages. In local areas with more than one server, it is important to divide the assignment of primary inboxes evenly among the servers. The division should be along organizational lines reflecting communication patterns and ease of administration. Assignment of secondary inbox sites is more difficult.
Occasionally, the user experience of Grapevine deviates from the conceptually image of service by a monolithic server. The most common cause for surprise is delays in propagating registration database changes. Symptoms include deleting names, administrations, duplicate messages. It is better to educate the users of such limitations.
In building Grapevine many design decisions were made based on assumptions about the nature of the expected load. Efficient operation of the system depended on the accuracy of these predictions. Because Grapevine is geographically dispersed, it is important for smooth and efficient operation to make monitoring, control and repair functions accessible through the internet.
A design objective of Grapevine is high reliability. A primary technique for achieving high reliability is replication of function among several servers. The system has done well in this regard. Aside from bug-free software, reliable hardware, and reliable communications being the foundations of a reliable system, appropriate management of resources is equally improtant. One violation of the availability principle is the backup operation of message to separate file servers. If a file sever is down then the messages stored on it become unavailable.
This archive was generated by hypermail 2.1.6 : Sat Jan 31 2004 - 17:44:23 PST