From: Steve Arnold (stevearn_at_microsoft.com)
Date: Tue Feb 24 2004 - 20:58:39 PST
Fox and company have designed a distributed system for managing services used by large numbers of users. It uses clustering as its method of scaling. The two cases they implemented are HotBot (the search engine) and TranSend, a web caching system for a dial-up pool.
The system is composed of several parts. There is the front-end, which is scalable, where most of the requests are handled and re-directed. There are the workers (or distillers) that perform a specific function. This are also scalable on the fly (except, interestingly, in the Hotbot case). Then there a few shared resources: the manager, user profile dB, and a monitor (along with the network).
Each element has a fairly simple interface. The manage monitors workloads (via some simple metric) and assign incoming requests accordingly. The whole system is based on BASE semantics, that is, everything is certainly not guaranteed to be correct, as contrasted to an ACID system. Most information is periodically multicasted, so it may not get updated right away.
The authors were able to create quite a lengthy paper without ever going into much detail. It's great that this level of research was going on in 1997, as the demand for such systems was only beginning. I would like to have read more about just how you could extend the system and more of the interface details, rather than there mere mentioning that it had been extended.
This archive was generated by hypermail 2.1.6 : Tue Feb 24 2004 - 20:58:35 PST