From: Cliff Schmidt (cliff_at_bea.com)
Date: Mon Feb 02 2004 - 16:04:55 PST
This paper describes one of the early large-scale Internet distributed
systems developed within Xerox PARC. Published in 1984, after a few
years of experience with the system, the paper touches on the high-
level design decisions, trade-offs, and lessons learned, most of which
could be (and have been?) applied to modern distributed systems,
including not just email servers, but also authentication servers,
provisioning services, and just about any distributed design.
Grapevine was divided into a message service and registration service.
As we've seen before, messaging and membership often turn out to be
interdependent problems. These services seems to have pioneered
the way to deal with many of the key challenges that affect email
servers today. For instance, scalable distribution lists that
allowed users to manage their memberships was an example of how
the system designers did not even understand what problems needed to
be solved until the system was updated. For instance, they designed
the system expecting that list memberships would reflect organization
membership, instead they found distribution list membership being
driven more by common interests, which grew in proportion to the size
of the community. The authors explain that with this knowledge they
would have added an additional level of indirection. Another
problem with membership lists they ran into was how to prevent users
from receiving duplicate messages that are sent to multiple membership
lists, of which they subscribe to. One answer is to expand the lists
and check for duplicates before sending, but this isn't always
possible when communication failures exist to the replicated
registration servers, and when it is possible it causes an expensive
and often needless delay.
One of Grapevine's goals was to deal with replication to improve
response time and availability. It was interesting to read how the
authors were having to learn, as the internet grew, that they no
longer could control, or even predict, the path of communication
between any two servers. This impacted their decisions on where to
place replicated registries relative to message servers and clients.
At one point in the paper, the authors observe that "Grapevine lacks
detailed knowledge about the properties of the internet...cannot take
into consideration the bandwidth, reliability, or congestion...".
The combination of addressing group membership lists and transparent
distributed replication, lead to other interesting problems that did
not appear to have been previously dealt with. Specifically, changes
in membership was causing inconsistencies across servers that had not
yet received the update; for instance, sending a mail to a distribution
list immediately after adding a new member may not include the member
if the change had not yet been propagated.
Some of the other interesting issues Grapevine raised include:
- the importance of providing system availability status to users
- registration services as authentication for file servers
- caching of authorization within file servers when using registration
services
- mail servers as buffers before off-loading onto clients, but also
as semi-permanent repositories as users chose to access mail from
multiple sites (such as from home with the growth of the PC market).
- System management with local operators and remote system experts
- remote dynamic monitoring and repair tools for system experts
- redundancy across servers and problems that occur when not everything
is redundant (such as their first attempt at archiving part of the
inbox onto only one server)
- problems with updating and documenting a very large piece of
software
This archive was generated by hypermail 2.1.6 : Mon Feb 02 2004 - 16:04:57 PST