CSE 599c Hot Topics in Data Management Systems

Lecture 11: Distributed Data Management

Magdalena Balazinska

Winter 2006

Outline

What we have seen so far

Where we are going next

Distributed data management

Traditional approaches (part 1)

Overview of limitations and challenges

Outline of what is to come

Previous Lectures: Background

Relational database management systems

Data model

Data storage and indexing

Query evaluation: optimization and execution

Active databases: constraints and triggers

Previous Lectures: Hot Topics

Stream processing & continuous monitoring

Continuously adaptive processing

Approximation

Load shedding and sampling

General problem: inaccuracies and uncertainty

Device heterogeneity

Sensor networks

Distributed processing

Placement problem and fault-tolerance

References for Today's Lecture

Content of today's lecture is based on

M. Tamer Ozsu and Patrick Vaduriez. Principles of distributed database systems. Second Ed. Prentice-Hall 1999.

Donald Kossmann. The State of the Art in Distributed Query Processing. ACM Computing Survey. 32(4): 422-469. 2000.

M. Stonebraker and J. Hellerstein ed. Readings in Database Systems (Red Book). Third ed. Introduction to chapter 4.

stored

data

Distributed DBMS Definition

Important: many forms and definitions

Our definition: shared nothing infrastructure

Multiple machines connected with a network

DBMS

stored

data

DBMS

stored

data

DBMS

stored

data

DBMS

Network

Why Distributed Data Managment?

Scalability (ex: Amazon, eBay, Google)

Many small servers cheaper than large mainframe

Need to scale incrementally

Inherent distribution

Large organizations have data at multiple locations (different offices) -> original motivation

Distributed apps: workflow mgt, collaborative work

Web-based and Internet-based applications

Integrating different types of data

Goals of a Distributed DBMS

Shield users from distribution details

Naming and location transparency

Performance transparency

distributed query optimization

Fragment and copy transparency

Transaction transparency

Traditional Distributed DBMS

70's and 80's, three main prototypes:

SDD-1, distributed INGRES, and R*

Main components of a distributed DBMS

Defining data placement and fragmentation

Distributed query optimization (today)

Distributed transactions (next two lectures)

Managing replicas (later)

Review: Query Evaluation

Query Parser

Query Rewrite

Optimizer

Executor

Steps of query evaluation

Reserves

Sailors

sid = sid

sbid=1 and rating > 5

psname

(File scan)

(File scan)

(Nested loop)

(On-the-fly)

(On-the-fly)

Query plan

Review: Query Optimization

Enumerate alternative plans

Many possible equivalent trees: e.g., join order

Many implementations for each operator

nested-loop join, sort-merge, different access paths

Compute estimated cost of each plan

Compute number of I/Os and CPU utilization

Based on statistics

Choose plan with lowest cost

Query optimization is a hard problem!

Distributed Query Optimization

Distributed optimization is even harder

Search space is larger: must also select sites

Minimize resource utilization

I/O, CPU, & communication costs

Example cost function used in R*

WCPU Nbinst + WI/O NbI/O + Wmsg Nbmsg + WbyteNbbytes

Could also try to minimize response time

Optimization in R*: Example

Join R and S (S is inner relation)

Join at site 1 --> ship S (whole or matches)

Join at site 2 --> ship R

Join at site 3 --> ship R and S

R

Site 1

S

Site 2

other

Site 3

Network

Traditional Approach Limitations

Top-down

Global, a priori data placement

Global query optimization, one query at a time

Distributed transactions (02/14)

Assumes full cooperation of all sites

Assumes uniform sites

Assumes short-duration activities

Limited scalability

Challenges

Autonomy: different administrative domains

Cannot always assume full cooperation (02/16)

Do not want distributed transactions (02/23)

Heterogeneity

Different capabilities at different locations (02/21)

Different data types (03/07 & 03/09)

Large-scale

Internet-scale query processor (02/28)

Fundamental concepts in replication (03/02)

2006-01-01T23:16:26