CSE 599c Hot Topics in Data Management Systems

Lecture 20: Unstructured Data and More...

Magdalena Balazinska

Winter 2006

Today's Paper

Sergey Brin and Lawrence Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. WWW7. 1998.

Content of today's lecture also from

Raghu Ramakrishnan and Johannes Gehrke. Database Management Systems. Third Ed.

Outline

A very different kind of data management

Information retrieval overview

Web search overview

Google architecture unlike that of a DBMS

Extending a DBMS with full text search

How about other types of data?

Information Retrieval (IR)

Goal: search collection of text documents

Field evolved independently of databases

Two types of queries

Boolean query: recipe AND (beef or stew)

Ranked query: “recipe beef stew”

Evaluation criteria

Precision: % retrieved docs relevant to query

Recall: % relevant docs in db that are returned

IR Framework

Term frequencies (TF) term j document i: tij

Inverse doc. frequency (IDF): log(N/nj)

wij = TF * IDF

Length normalization: wij* = wij / ( wik2)

Document vector

Document Similarity and Ranking

Assume a t-dimensional space (t is nb terms)

Similarity is a dot-product of doc vectors

sim(Q, Di) = terms qj* wij*

Indexing Text Documents

Inverted index

Maps each word to a list of docs that contain it

B+ tree or hash index (lexicon) on all terms

Use inverted index for queries

Boolean query: intersect/merge doc lists

Ranked query: compute relevance of every doc with respect to query terms

Web Search

Goal: high quality results

Challenges

Large number of documents

Large number of terms

Malicious content publishers

Users are only willing to look at a few results

Observations

Additional info in hyperlink graph

Associate anchor text with destination page

Web Search Basic Idea

Crawl the Web and store each page

Build data structures

Compute PageRank for each page

Index all pages (forward and inverted index)

Produce a lexicon

Answer queries using the data structures

Google Data Structures

Figure 3 from

[brin1998]

Ranking Query Results

PageRank gives a page's “importance”

PR(A) = (1-d) + d (PR(T1)/C(T1)+...+ PR(Tn)/C(Tn))

d: damping factor

T1 through Tn are the citations for this page

C(A): number of outgoing links from page A

To prioritize search results also use IR score

type-weights: title, anchor, URL, big font, etc.

count-weights: how many hits of each type

type-proxy-weights: distance between words

Google Architecture

Figure 1 from

[brin1998]

DBMS Architecture

Process Manager

Admission Control

Connection Mgr

Query Processor

Parster

Query Rewrite

Optimizer

Executor

Storage Manager

Access Methods

Lock Manager

Buffer Manager

Log Manager

Shared Utilities

Memory Mgr

Disk Space Mgr

Replication Services

Admin Utilities

[Anatomy of a Database System.

J. Hellerstein and M. Stonebraker.

Red Book. 4ed.]

Text Search in a DBMS

SQL standard: new data type FullText

Incorporate keyword search into SQL

Also support “rank” method

Typical implementation

Separate, loosely-coupled text-search engine

Update index asynchronously on inserts

No transactional guarantees

Text Search in DB2

Text Extender (TE)

Text indexes on DB2 tables

Relate index entries back to individual records

Triggers keep text and text indexes in synch

More info: [A. Maier and D. Simmen. DB2 Optimization in Support of Full Text Search]

General Heterogeneity Problem

Typical DBMS solution

Object-relational: support for user-defined types

But often need native support for performance

Specialized tools: i.e., video db, image db, geographic db. Good for one type at the time

Google also focuses on specialized tools

Are specialized tools the answer?

How about federated search across all types?

How about providing more than search?

Conclusion

Offering data management services over heterogeous types of data is a hard problem

In general, there are many open problems in the field of data management

Growing popularity of Web and Internet

Opportunities and challenges

Pervasice and mobile devices

Also create interesting problems and challenges

Hope you enjoyed the class!

2006-01-01T23:16:26