CSE 163, Spring 2020: Homework 2: Processing CSV Data

Overview

In this assignment, you will use the foundational Python skills you've been developing and apply them to analyze a small dataset. Many datasets you’ll be working with are structured as CSV or tabular representation - this assignment will be an introduction to reading, processing, and grouping rows and columns to calculate some interesting statistics. These skills will be very useful to have a strong foundation in when we work with much larger (and less complete) real-world datasets!

This assignment is broken in to two main parts, where each part mostly does the same computations in different ways. This is to give you the opportunity to compare/contrast different approaches to solving problems.

Files

If you are developing on Ed, all the files are there. If you are developing locally, you should download the starter code hw2.zip and open it as the project in Visual Studio Code. The files included are:

• hw2_manual.py: The file for you to put solutions to Part 0.
• hw2_pandas.py: The file for you to put solutions to Part 1.
• hw2_test.py: The file for you to put your tests for Part 0 and Part 1.
• cse163_utils.py: A file where we will store utility functions for helping you write tests.
• run_hw2.py: A client program provided to call your functions. This is just for your convenience.
• pokemon_box.csv: A CSV file that stores information about Pokemon. This columns of this file are explained below.
• pokemon_test.csv: A very small CSV file that stores information about Pokemon. This columns of this file are explained below.

Data

For this assignment, you will be working with a dataset of Pokemon that you have caught on your Pokemon journey so far. The file pokemon_box.csv stores all the data about the captured Pokemon and has a format that looks like:

Note that because this is a CSV file, the file contents have these cells separated by commas.

Column Descriptions

• id: Unique identification number corresponding to the species of a Pokemon. Note that if there are multiple Pokemon of the same species in the dataset, they all share the id.
• name: Name of the species of Pokemon. For example Pikachu.
• level: The level of this Pokemon (an integer)
• personality: A one-word string describing the personality of this Pokemon
• type: A one-word string describing the type of the Pokemon (e.g. "Grass" for Bulbasaur)
• weakness: What type this Pokemon is weak to. For example, Bulbasaur is considered weak to the fire type.
• atk, def, hp: Pokemon stats that indicate how many hits a Pokemon can take (hp), how strong its attacks are (atk), and how much hits affect it (def)
• stage: Indicates if this Pokemon has evolved into a new species. For example, in the Charmander species (stage 1), it evolves into a Charmeleon (stage 2), which evolves into Charizard (stage 3).

Part 0

In this part of the homework, you will write code to perform various analytical operations on data parsed from a file into the list of dictionaries represenation.

For this step of the assignment, you will be implementing various functions to answer questions about the dataset.

Each function should take the list returned by the cse163_utils.parse function (provided for you) as the first argument, along with any other arguments specified in each problem. For example, for the third function, we would call filter_range(data, 1, 10) where data was the list returned by cse163_utils.parse.

This data structure should not be modified by any function you write. Every problem that deals with strings should be case-sensitive (this means "chArIzard" is a different species than "Charizard"). You may make the following assumptions about the inputs:

• You may assume the given list is non-empty for all functions you implement.
• For each problem, you may assume we pass parameters of the expected types described for that problem and that those parameters are not None.
• You should make no other assumptions about the parameters or the data.

For each of the problems, we will use the file pokemon_test.csv to show what should be returned.

id,name,level,personality,type,weakness,atk,def,hp,stage
59,Arcanine,35,impish,fire,water,50,55,90,2
59,Arcanine,35,gentle,fire,water,45,60,80,2
121,Starmie,67,sassy,water,electric,174,56,113,2
131,Lapras,72,lax,water,electric,107,113,29,1

species_count(data)  # 3

max_level(data)  # ('Lapras', 72)

filter_range(data, 30, 70)  # ['Arcanine', 'Arcanine', 'Starmie']

mean_attack_for_type(data, 'fire')  # 47.5

count_types(data)  # {'water': 2, 'fire': 2}

highest_stage_per_type(data)  # {'water': 2, 'fire': 2}

mean_attack_per_type(data)  # {'water': 140.5, 'fire': 47.5}

Part 1

In this part of the homework, you will write code to do various analytical operations on CSV data. Sounds a bit like Déjà vu! The functions you write for this assignment are exactly the same, but instead you will be using the pandas library to solve the functions.

For this step of the assignment, you will be implementing various functions to answer questions about the dataset. All of the problems are the same as Part 0 save for the fact that the input is now a DataFrame; you can check back to that page to see any examples you might need for this part.

Each function should take the DataFrame returned by the pd.read_csv function as the first argument, along with any other arguments specified in each problem. For example, for the third function, we would call filter_range(data, 1, 10) where data was the DataFrame returned by pd.read_csv.

This data structure should not be modified by any function you write. Every problem that deals with strings should be case-sensitive (this means "chArIzard" is a different species than "Charizard"). You may make the following assumptions about the inputs:

• You may assume the DataFrame is non-empty for all functions you implement.
• For each problem, you may assume we pass parameters of the expected types described for that problem and that those parameters are not None.
• You should make no other assumptions about the parameters or the data.

name,age,species
Fido,4,dog
Meowrty,6,cat
Chester,1,dog
Phil,1,axolotl

names = data['name']  # Series
list(names)  # ['Fido', 'Meowrty', 'Chester', 'Phil']
dict(names)  # {0: 'Fido', 1: 'Meowrty', 2: 'Chester', 3: 'Phil'}
row = data.loc[1]  # Series
list(row)  # ['Meowrty', 6, 'cat']
dict(row)  # {'name': 'Meowrty', 'age': 6, 'species': 'cat'}

Part 2

In this part of the assignment, you will write tests for your solutions in Part 0 and Part 1.

Like in Homework 1, we have provided a function called assert_equals that takes an expected value and the value returned by your function, and compares them: if they don't match, the function will crash the program and tell you what was wrong. You can see more instructions an example for tests from the Homework 1 - Part 1 to see examples of how to call the tests.

Part 2 Expectations

For full credit, your hw2_test.py must satisfy all of the following conditions:

• You should not use the file pokemon_box.csv to count for your own test cases. The file is too large to be able to meaningfully come up with the correct answer on your own (e.g., it's not valid to run your code and then paste the output as the "correct output"). You should submit your own testing CSV files.
• Uses the main method pattern shown in class.
• Your tests should be efficient in the sense that you don't have to re-read the datasets from the CSV for every tests function. If you want to use the same dataset in multiple functions, read it in once in main and pass them around as parameters to your test functions.
• Has a function to test each of the functions in the hw2_manual.py and hw2_pandas.py that you were asked to write. It's okay to merge the test functions for the same problem in the different parts of the homework since their outputs are the same. You should not organize your tests such that there are only two test functions (e.g., test_hw2_manual and test_hw2_pandas) as we want to separate the test functions by each of the problems in the spec. There might be some redundancy in your tests which is expected since it's hard to factor out the different function calls and input types.
• For each function you need to test, you should include a test that tests the example from the spec and one additional test that is different from what is shown in the spec. A single test is considered a call on assert_equals.
• Each of these test functions should have a descriptive name that indicates which function is being tested (e.g. test_funky_sum)
• Each of the test functions must be called from main.
• Turn in any test CSV files you generate.

Evaluation

Your submission will be evaluated on the following dimensions:

• Your solution correctly implements the described behaviors. You will have access to some tests when you turn in your assignment, but we will withhold other tests to test your solution when grading. All behavior we test is completely described by the problem specification or shown in an example.
• When we run your code, it should produce no errors or warnings.
• You should remove any debug print statements. Some students report Ed crashing when they try to print the whole dataset; remove those extra print statements to prevent excessive output.
• Your code meets our style requirements:
  • All files submitted pass flake8
  • Your program should be written with good programming style. This means you should use the proper naming convention for methods (snake_case), your code should not be overly redundant and should avoid unnecessary computations.
  • Every function written is commented using a doc-string format that describes its behavior, parameters, returns, and highlights any special cases.
  • There is a comment at the top of each file you write with your name, section, and a brief description of what that program does.
  • Any expectations on this page or the sub-pages for the assignment are met as well as all requirements for each of the problems are met.

Submission

This assignment is due by Thursday, April 23 at 23:59 (PDT). Please refer to the late policy for the policy on how late you may turn an assignment in.

You should submit your finished hw2_manual.py, hw2_pandas.py, and hw2_test.py on Ed.

You may submit your assignment as many times as you want before the late cutoff (remember submitting after the due date will cost late days). Recall on Ed, you submit by pressing the "Mark" button. You are welcome to develop the assignment on Ed or develop locally and then upload to Ed before marking.