Assignment 2: Exploratory Data Analysis
In this assignment, you will identify a dataset of interest and perform an exploratory analysis to better understand the shape & structure of the data, investigate initial questions, and develop preliminary insights & hypotheses. Your final submission will take the form of a report consisting of captioned visualizations that convey key insights gained during your analysis.
Step 1: Data Selection
First, you will pick a topic area of interest to you and find a dataset that can provide insights into that topic. To streamline the assignment, we've pre-selected a number of datasets included below for you to choose from.
However, if you would like to investigate a different topic and dataset, you are free to do so. If working with a self-selected dataset, please check with the course staff to ensure it is appropriate for the course. Be advised that data collection and preparation (also known as data wrangling) can be a very tedious and time-consuming process. Be sure you have sufficient time to conduct exploratory analysis, after preparing the data.
After selecting a topic and dataset – but prior to analysis – you should write down an initial set of at least three questions you'd like to investigate.
Part 2: Exploratory Visual Analysis
Next, you will perform an exploratory analysis of your dataset using a visualization tool such as Tableau. You should consider two different phases of exploration.
In the first phase, you should seek to gain an overview of the shape & stucture of your dataset. What variables does the dataset contain? How are they distributed? Are there any notable data quality issues? Are there any surprising relationships among the variables? Be sure to also perform "sanity checks" for patterns you expect to see!
In the second phase, you should investigate your initial questions, as well as any new questions that arise during your exploration. For each question, start by creating a visualization that might provide a useful answer. Then refine the visualization (by adding additional variables, changing sorting or axis scales, filtering or subsetting data, etc.) to develop better perspectives, explore unexpected observations, or sanity check your assumptions. You should repeat this process for each of your questions, but feel free to revise your questions or branch off to explore new questions if the data warrants.
Your final submission should take the form of a report – similar to a slide show or comic book – that consists of 10 or more captioned visualizations detailing your most important insights. Your "insights" can include important surprises or issues (such as data quality problems affecting your analysis) as well as responses to your analysis questions. To help you gauge the scope of this assignment, see this example report analyzing data about motion pictures.
Each visualization image should be a screenshot exported from a visualization tool, accompanied with a title and descriptive caption (1-4 sentences long) describing the insight(s) learned from that view. Provide sufficient detail for each caption such that anyone could read through your report and understand what you've learned. You are free, but not required, to annotate your images to draw attention to specific features of the data. You may perform highlighting within the visualization tool itself, or draw annotations on the exported image. To easily export images from Tableau, use the Worksheet > Export > Image... menu item.
To help formulate your report, we've provided a basic HTML template for you to fill in. You will need to edit the HTML to add your captions and links to image files. Exported image files should reside in the same local directory as your HTML file. The end of your report should include a brief summary of main lessons learned. When complete, submit a zip file to Canvas with your HTML file and visualization screenshot images.
Recommended Data Sources
To get up and running quickly with this assignment, we recommend exploring one of the following provided datasets.
The World Bank Data, 1960-2017
The World Bank has tracked global human development by indicators such as climate change, economy, education, environment, gender equality, health, and science and technology since 1960. We have 20 indicators from the World Bank for you to explore. Alternatively, you can browse the original data by indicators or by countries. Click on an indicator category or country to download the CSV file.
Campaign Finance Data for the 2017-18 Congressional Cycle
Congressional elections will be held November 2018 with political fundraising and spending well underway. The Federal Election Commission (FEC) maintains a database of the financial activity of Political Action Committees (PACs), including funds raised, money disbursed to committees & candidates, and operating expenditures. See the FEC site for a full listing of available data sets.
Note that these data sets are large, with both many rows and many columns. It is sufficiently rich to support a variety of explorations. If you choose to explore this data, we strongly encourage you to pick a focus area. Examples include: (1) How do presidential campaigns continue to spend money after the general election? (2) How are the primary Democratic and Republican National Committees spending funds? (3) What are Democratic Party presidential hopefuls doing? (4) What expenditures are occurring across Washington State, and by whom? (5) To what degree is money spent back in politician's home states? To what extent does it go to Beltway consultants? Focusing on any one of these topics should be sufficient for this assignment.
Chicago Crimes, 2001-Present
This dataset reflects reported incidents of crime (with the exception of murders where data exists for each victim) that occurred in the City of Chicago from 2001 to present, minus the most recent seven days. Data is extracted from the Chicago Police Department's CLEAR (Citizen Law Enforcement Analysis and Reporting) system.
Daily Weather in the U.S., 2017
This dataset contains daily U.S. weather measurements in 2017, provided by the NOAA Daily Global Historical Climatology Network. This data has been transformed: some weather stations with only sparse measurements have been filtered out. See the accompanying weather.txt for descriptions of each column.
Yelp Open Dataset
This dataset provides information about businesses, user reviews, and more from Yelp's database. The data is split into separate files (business, checkin, photos, review, tip, and user), and is available in either JSON or SQL format. You might use this to investigate the distributions of scores on yelp, look at how many reviews users typically leave, or look for regional trends about restaurants. Note that this is a large, structured dataset and you don't need to look at all of the data to answer interesting questions.
In order to download the data you will need to enter your email and agree to Yelp's Dataset License.
Additional Data Sources
If you want to investigate datasets other than those recommended above, here are some possible sources to consider. You are also free to use data from a source different from those included here. If you have any questions on whether your dataset is appropriate, please ask the course staff ASAP!
- data.seattle.gov - City of Seattle Open Data
- data.wa.gov - State of Washington Open Data
- nwdata.org - Open Data & Civic Tech Resources for the Pacific Northwest
- data.gov - U.S. Government Open Datasets
- U.S. Census Bureau - Census Datasets
- IPUMS.org - Integrated Census & Survey Data from around the World
- Federal Elections Commission - Campaign Finance & Expenditures
- Federal Aviation Administration - FAA Data & Research
- fivethirtyeight.com - Data and Code behind the Stories and Interactives
- Buzzfeed News
- Socrata Open Data
- 17 places to find datasets for data science projects
You are free to use one or more visualization tools in this assignment. However, in the interest of time and for a friendlier learning curve, we strongly encourage you to use Tableau. Tableau provides a graphical interface focused on the task of visual data exploration. You will (with rare exceptions) be able to complete an initial data exploration more quickly and comprehensively than with a programming-based tool.
- Tableau - Desktop visual analysis software. Available for both Windows and MacOS; register for a free student license.
- Voyager - Research prototype from the UW Interactive Data Lab. Voyager combines a Tableau-style interface with visualization recommendations. Use at your own risk!
- R, using the ggplot2 library or with R's built-in plotting functions.
- Jupyter Notebooks (Python), using libraries such as Altair or Matplotlib.
Data Wrangling Tools
The data you choose may require reformatting, transformation or cleaning prior to visualization. Here are tools you can use for data preparation. We recommend first trying to import and process your data in the same tool you intend to use for visualization. If that fails, pick the most appropriate option among the tools below. Contact the course staff if you are unsure what might be the best option for your data!
- Tableau - Tableau provides basic facilities for data import, transformation & blending.
- Trifacta Wrangler - Interactive tool for data transformation & visual profiling.
- OpenRefine - A free, open source tool for working with messy data.
Each submission will be graded based on both the analysis process and included visualizations. Here are our grading criteria:
- Poses clear questions applicable to the chosen dataset.
- Appropriate data quality assessment and transformation.
- Sufficient breadth of analysis, exploring multiple questions.
- Sufficient depth of analysis, with appropriate follow-up questions.
- Expressive & effective visualizations crafted to investigate analysis questions.
- Clearly written, understandable captions that communicate primary insights.
This is an individual assignment. You may not work in groups.
Your completed exploratory analysis report is due Tuesday 10/16, 11:59pm. You should fill out the provided HTML template and upload to Canvas a zip file containing your completed HTML file and visualization screenshots.