Machine Learning using scikit-learn
Now you will be making a simple machine learning model for the provided education data using scikit. Complete this in a function called fit_and_predict_degrees
that takes the data as a parameter and returns the test mean square error as a float. This may sound like a lot, so weβve broken it down into four steps for you:
- Filter the
DataFrame
to only include the columns for year, degree type, sex, and total. Drop any rows that have missing data for these columns and then convert string values to their dummy encoding. Split the columns as needed into input values and target values.
- Split the dataset into 80% for training and 20% for testing.
- Train a decision tree regressor model to take in year, degree type, and sex to predict the percent of individuals of the specified sex to achieve that degree type in the specified year.
- Use your model to predict on the test set. Calculate the accuracy of your predictions using the mean squared error of the test dataset.
You do not need to anything fancy like find the optimal settings for parameters to maximize performance. We just want you to start simple and train a model from scratch!
You can find our reference sheet for machine learning with scikit-learn here. This reference sheet has information about general scikit-learn calls that are helpful, as well as how to train the tree models we talked about in class.
Development Strategy
Like in Part 1, it can be difficult to write tests for this section. Because of this, weβve provided some alternative ways to gain confidence in your result:
- Print your test y values and your predictions to compare them manually. They wonβt be exactly the same, but you should notice that they have some correlation. For example, I might be concerned if my test y values were [2, 755, β¦] and my predicted values were [1022, 5...] because they seem to not correlate at all.
- Calculate your mean square error on your training data as well as your test data. The error should be lower on your training data than on your testing data.