Lecture 12. Parallel data processing

Readings:

There are three papers for this lecture, but we only ask you to read a few subsections from each. If it still takes you more than three hours, scheme some of the sections quickly. Focus on the key ideas.

• Dave DeWitt and Jim Gray. Parallel Database Systems: The Future of High Performance Database Systems. Communications of the ACM. 1992. Also in Red Book 4th Ed. Sections 1 and 2 only. [pdf]
• Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. OSDI 2004. Focus on sections 1 through 4. [pdf]
• C. Olston, B. Reed, U. Srivastava, R. Kumar and A. Tomkins. Pig Latin: A Not-So-Foreign Language for Data Processing. SIGMOD 2008. Only the introduction. [pdf]

As you read these papers, discuss some of the similarities and differences between a parallel DBMS and Pig/MapReduce.

Lecture notes:

• lecture12.pdf

Additional resources:

• Chapter 22 (in R&G, third edition).

Optional, additional readings:

• Related to the Dryad project at Microsoft Research
  • Dryad and DryadLINQ project websites (include papers).
  • SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. Ronnie Chaiken, Bob Jenkins (Microsoft), Paul Larson(Microsoft Research, USA), Bill Ramsey, Darren Shakib, Simon Weaver (Microsoft), Jingren Zhou (Microsoft Research, USA). VLDB 2008.
• Related to MapReduce
  • Interpreting the Data: Parallel Analysis with Sawzall. Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan. Scientific Programming Journal
    Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure 13:4, pp. 227-298.
  • Hadoop project website
  • Map-reduce-merge: simplified relational data processing on large clusters. Hung-chih Yang (Yahoo!), Ali Dasdan (Yahoo!), Ruey-Lung Hsiao (UCLA), D. Stott Parker (UCLA). SIGMOD 2007.
• Pig project website at Yahoo! Research.
• Clustera: An Integrated Computation and Data Management System. David DeWitt, Eric Robinson, Srinath Shankar, Erik Paulson, Jeffrey Naughton, Andrew Krioukov, Joshua Royalty (UW - Madison). VLDB 2008