{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Homework 2: DNA analysis\n", "\n", "**Due: at 11pm on Wednesday, Jan 23, 2019.**
\n", "Submit via Canvas. \n", "\n", "## Learning Objectives:\n", "\n", "- Gain experience writing Python code using loops, conditionals (if statements), functions and string manipulation\n", "- Become familiar with running Python programs from the command line and using command line parameters to locate data files\n", "- Write Python code to analyze DNA data sets\n", "\n", "**NOTE:** Although you will fill in the body of one function and call it, you do not need to write additional functions for this assignment. You are encouraged to examine the provided function filename_to_string, which opens and reads in data files, but you do not need to understand anything about opening and reading files in order to do this assignment. We have already covered everything you need to know to be able to complete this entire assignment. For all of our assignments (except your final project) you should **NOT** use parts of Python not yet discussed in class or the course readings.\n", "\n", "Advice from previous students about this assignment: [14wi 15sp](https://canvas.uw.edu/courses/1254701/assignments/4615077)\n", "\n", "This assignment is posted on [Canvas](https://canvas.uw.edu/courses/1254701/assignments/4615077)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Background\n", "\n", "You will use, modify, and extend a program to compute the GC content of DNA data. The GC content of DNA is the percentage of nucleotides that are either G or C.\n", "\n", "DNA can be thought of as a sequence of nucleotides. Each nucleotide is adenine, cytosine, guanine, or thymine. These are abbreviated as A, C, G, and T. A nucleotide is also called a nucleotide base, nitrogenous base, nucleobase, or just a base.\n", "\n", "Biologists have multiple reasons to be interested in GC content:\n", "\n", "GC content can identify genes within the DNA, and can identify types of genes. Genes tend to have higher GC content than other parts of the DNA. Genes with longer coding regions have even higher GC content.\n", "Regions of DNA with higher GC content require higher temperatures for certain chemical reactions, such as when copying/duplicating the DNA.\n", "\n", "GC content can be used in determining classification of species.\n", "\n", "If you are curious, Wikipedia has more information about GC content. That reading is optional and is not required to complete this assignment.\n", "\n", "Your program will read data files produced by a high-throughput sequencer — a machine that takes as input some DNA, and produces as output a file containing a sequence of nucleotides.\n", "\n", "Here are the first 8 lines of output from a particular sequencer:\n", "\n", "```\n", "@SOLEXA-1GA-2_2_FC30DNN:1:2:574:1722\n", "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC\n", "+SOLEXA-1GA-2_2_FC30DNN:1:2:574:1722\n", "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh\n", "@SOLEXA-1GA-2_2_FC30DNN:1:2:478:1745\n", "GTGGGGGTGATGTCCACGATTACGCCGACCGGCTGG\n", "+SOLEXA-1GA-2_2_FC30DNN:1:2:478:1745\n", "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh\n", "```\n", "\n", "The nucleotide data is in the second line, the sixth line, the tenth line, etc. Your program will not use the rest of the file, which provides information about the sequencer and the sequencing process that created the nucleotide data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 1: Obtain the files, add your name\n", "\n", "Obtain the files you need by downloading the homework2.zip file. (This is a large download — be patient.)\n", "\n", "Unzip the homework2.zip file to create a homework2 directory/folder. You will do your work here. The homework2 directory/folder contains:\n", "\n", "- *dna_analysis.py*, a partial Python program that you will complete\n", "- *answers.txt*, a file where you will answer textual questions\n", "- data, a directory. Which contains the data that you will process:\n", " - *.fastq files, which are output from DNA sequencers; this is the data that the program analyzes\n", " - expected_output, a directory containing example runs of the final result of your *dna_analysis.py* program.\n", " \n", "You will do your work by modifying two files — __*dna_analysis.py*__ and __*answers.txt*__ — and then submitting the modified versions. **Add your name** to the top of each of these files.\n", "\n", "Each problem will ask you to make some changes to the program *dna_analysis.py* (or to write text in the *answers.txt* file, or both). When you do so, you will generally add to the program. Do not remove changes from earlier problems when you work on later problems; your final program should solve all the problems.\n", "\n", "In either file, keep the number of characters on a particular line below 80, otherwise your files become hard to read. One technique to do this in Python is to break large equations into smaller ones by storing subexpressions in variables. Breaking things down into smaller computations can also help with debugging." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By the end of the assignment, we would like __*dna_analysis.py*__ to produce output of the **exact** form:\n", "\n", "```\n", "GC-content: ___\n", "AT-content: ___\n", "G count: ___\n", "C count: ___\n", "A count: ___\n", "T count: ___\n", "Sum of G+C+A+T counts: ___\n", "Total count: ___\n", "Length of nucleotides: ___\n", "AT/GC Ratio: ___\n", "GC Classification: ___\n", "```\n", "\n", "Where ___ is replaced by values that you will calculate. Of course, the exact values in each category will vary depending on the input data that you are using. We expect the formatting of your program output to **exactly match** this." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tips\n", "\n", "**Check your output** - You should validate your dna_analysis.py program's output using the Diff Checker before submitting your assignment. You can compare your output to the files given in the expected_output directory of the homework2 files. Drag a file to a window in the Diff Checker. Select text in a command prompt window and paste it into the other window and select \"Find Difference!\".\n", "\n", "**Editing text files** - You will submit **answers.txt** as a text file. Plain text is the standard for communicating information among programmers, because it can be read on any computer without installing proprietary software. You can edit text files using any text editor. If you use a word processor, then be sure to save the files as text. Windows users should never use Notepad, as Notepad will mangle the line endings in the file; WordPad or Notepad++ are better alternatives.\n", "\n", "**Command Prompt** - In the past some students have had trouble running from the command line when folder names had spaces in them. You may find it easiest to avoid this (that is, rename your folders so they do not have spaces in the name). \n", "\n", "To cut and paste things from the Windows command prompt, left click and drag the mouse to select text from anywhere in the command prompt window. When you have highlighted what you want, hit return. Now you can use control-V to paste into [Diff Checker](https://www.diffchecker.com/diff) or a text file.\n", "\n", "**You do not need to modify** *answers.txt* **or** *dna_analysis.py* **for this problem.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 2: Run the program\n", "\n", "When writing programs that analyze data (or any other type of program) it is important to check the correctness of your programs. One way to do this is by comparing the output of your program to a computation done in some other way, such as by hand or by a different program. We have provided the *test-small.fastq* file for this purpose. This file is small enough that you can easily open it up in a text editor and calculate the GC content by hand. Then, run your program to verify that it provides the correct answer for this file.\n", "\n", "You should be able to open up *test-small.fastq* and other input and output files in a text editor of your choice (although just clicking on a .fastq file is not likely to work since the .fastq file extension is not one that is known to your operating system). \n", "\n", "\n", "\n", "(If you are interested, sample_3.fastq and sample_4.fastq are from [Streptococcus pneumoniae TIGR4](https://en.wikipedia.org/wiki/Streptococcus_pneumoniae), and sample_5.fastq is from [Human herpesvirus 5](https://en.wikipedia.org/wiki/Cytomegalovirus).)\n", "\n", "If you have already used the [Diff Checker](https://www.diffchecker.com/diff), you might notice that some of your results are different than the example results. Don't worry about this — this issue will be resolved in Problem 6.\n", "\n", "Cut and paste the line of output produced by your program regarding GC-content when run on *sample_1.fastq* into your answers.txt file. See \"Tips\" at the top of this page for info on cutting and pasting things from the Windows command prompt. (Note, this could take a minute or so to run.) For example, your answer might look like:\n", "\n", "GC-content: 0.42900139393
\n", "(**Note:** This is not the answer you should expect to get; this is just an example of the format that your answer should be in.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 3: Remove some lines\n", "\n", "In your program, comment out this line:\n", "```\n", " gc_count = 0\n", "```\n", "by prefixing it by the # character. Save the file and then re-run the program from the comand line, just as you did for Problem 2. \n", "\n", "**In** *answers.txt*, **explain what happened, and why it happened.**\n", "\n", "Now, restore the line to its original state by removing the # that you added. What would happen if you commented out this line instead? (Feel free to try it!)\n", "```\n", " nucleotides = filename_to_string(filename)\n", "```\n", "\n", "**Explain what happens and why in** *answers.txt.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 4: Compute AT content\n", "\n", "Augment your program so that, in addition to computing and printing the GC ratio, it also computes and prints the AT content. The AT content is the percentage of nucleotides that are A or T.\n", "\n", "Two ways to compute the AT content are:\n", "\n", "- Copy the existing loop that examines each nucleotide and modify it. You will now have two loops, one of which computes the GC count and one of which computes the AT count. OR\n", "- Add more statements into the existing loop, so that one loop computes both the GC count and the AT count.\n", "\n", "You may use whichever approach you prefer. Add whatever new variables you need.\n", "\n", "Check your work by manually computing the AT content for file test-small.fastq, then comparing it to the output of running your program on test-small.fastq.\n", "\n", "Run your program on *sample_1.fastq*. Cut-and-paste the relevant line of output into **_answers.txt_**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 5: Count nucleotides\n", "\n", "Augment your program so that it also computes and prints:\n", "- the number of __A__ nucleotides \n", "- the number of **T** nucleotides\n", "- the number of __G__ nucleotides\n", "- the number of **C** nucleotides. \n", "\n", "Add whatever new variables you need.\n", "\n", "When doing this, add **at most one extra loop** to your program. You can solve this part without adding any new loops at all, by reusing an existing loop. At this point you should also feel free to modify the code we have given you if another structure of *if statements* makes more sense to you. We just caution you against looping through the data more times than you need to as this could cause your code to run **very slowly**.\n", "\n", "Check your work by manually computing the results for file *test-small.fastq*, then comparing them to the output of running your program on *test-small.fastq*.\n", "\n", "Run your program on *sample_1.fastq*. Cut-and-paste the relevant lines of output into **_answers.txt_** (the lines that indicate the G count, C count, A count, and T count)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 6: Sanity-check the data\n", "\n", "For each of the eleven .fastq files you have been given, calculate and print the following three quantities:\n", "\n", "- the sum of: the **A count, the C count, the G count, and the T count** (store this in a new variable called **sum_counts**)\n", "- the total_count variable (total number of nucleotides)\n", "- the length of the nucleotides string variable. You can compute this with len(nucleotides).\n", "\n", "In other words, compute the three quantities for _test-small.fastq_ and then do the same for _test-high-gc-1.fastq_, etc.\n", "\n", "For at least one .fastq file, at least one of these quantities will be different from the other two.
\n", "In your **_answers.txt_** file, state which .fastq file(s) and which quantities differ. (If all three quantities are equal for each .fastq file, then your code contains a mistake.) In your **_answers.txt_** file, write a short paragraph that explains why these differ.\n", "\n", "Explaining why (or debugging your code if all three quantities were the same in all .fastq files) might require you to do some detective work.\n", "\n", "This exercise is meant to expose you to a **situation you might encounter when processing a data file of your own** (say on your Final project). When your program does not give the results you expect, there are two likely sources of the problem. **One is that your program contains a bug!** Check your code carefully to be sure you are calculating all values correctly. We will talk about testing in more detail later but for now, try walking through your code with a very small data set and calculating values by hand. A second source of unexpected results that is very common with data files is that there is **something you were assuming about the contents of the data files that was an incorrect assumption**. This could include things like assuming each line would contain a certain number of characters or words, or that all characters would be uppercase or lowercase, or that values might only be in a certain range. If you wrote your program assuming something about your data files that was not correct, your program may not give correct results.\n", "\n", "To track down a wrong assumption about a data file, think about ways you can modify your program to help you determine what is happening. This could include having it **print out values** when they do not meet some asumption you are making about the file. You could also try just loading a data file into a **text editor** and examining it with your eyes to see if you see something you did not expect. (Although if you try this approach we strongly suggest that you start with the **smallest data file** for which the three quantities are not all the same.) Another approach would be to **modify your program, or create a new program**, to compute the three quantities for each line of a data file separately (as opposed to for the file as a whole as you have been doing): if the quantities differ for an entire file, then they must differ for at least one specific line in that file. Examining that/those line(s) will help you understand the problem.\n", "\n", "If all of the three quantities that you measured in problem 6 are the same, then it would not matter which one you used in the denominator when computing the GC content. However, you saw that the three quantities are not all the same. In **_answers.txt_**, state which of these quantities should be used in the denominator and which should not, and why.\n", "\n", "If your program incorrectly computed the GC content (which should be equal to (G+C)/(A+C+G+T)), then state that fact in your **_answers.txt_** file. Then, go back and correct your program, and also update any incorrect answers elsewhere in your **_answers.txt_** file. It is fine to change the code we provided you if needed.\n", "\n", "If you are unsure if you are calculating things correctly, now would be a good time to validate your _dna_analysis.py_ program's output using the [Diff Checker](https://www.diffchecker.com/diff). You can compare your output to the files given in the expected_output directory of the homework2 files. You have not yet completed the assignment, so your output will not be identical. But things like GC-content, AT-content and individual counts should be identical. You will produce the last two lines of output in the expected_output files in Problem 7 and Problem 8 below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 6" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 7: Compute the AT/GC ratio\n", "\n", "Sometimes biologists use the __AT/GC ratio__, defined as (A+T)/(G+C), rather than the GC-content, which is defined as (G+C)/(A+C+G+T).\n", "\n", "Modify your program so that it also computes the AT/GC ratio.\n", "\n", "Check your work by manually computing the results for file _test-small.fastq_.
\n", "Compare them to the output of running your program on _test-small.fastq_.\n", "\n", "Run your program on _sample_1.fastq_. Cut-and-paste the relevant lines of output into **_answers.txt_** (the line that indicates the AT/GC ratio).\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 7" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 8: Categorize organisms\n", "\n", "The GC content can be used to categorize microorganisms.\n", "Fill in the body of the classify function to return the classification of the organism (\"high\", \"moderate\" or \"low\") described in the data file given using these classifications:\n", "\n", "If the GC content is above 60%, the organism is considered “high GC content”.\n", "If the GC content is below 40%, the organism is considered “low GC content”.\n", "Otherwise, the organism is considered “moderate GC content”.\n", "\n", "Biologists can use GC content for classifying species, for determining the melting temperature of the DNA (useful for both ecology and experimentation, for example PCR is more difficult on organisms with high GC content), and for other purposes. Here are some examples:\n", "\n", "The GC content of Streptomyces coelicolor A3(2) is 72%.\n", "The GC content of Yeast (Saccharomyces cerevisiae) is 38%.\n", "The GC content of Thale Cress (Arabidopsis thaliana) is 36%.\n", "The GC content of Plasmodium falciparum is 20%.\n", "\n", "Again, test that your program works on some data files with known outputs. The test-small.fastq file has low GC content. We have provided four other test files, whose names explain their GC content: *test-moderate-gc-1.fastq*, *test-moderate-gc-2.fastq*, *test-high-gc-1.fastq*, *test-high-gc-2.fastq*.\n", "\n", "You will find a \"skeleton\" or \"stub\" for the classify function near the top of dna_analysis.py, just before where the main program begins. There is an assignment statement inside of the function body that is there only as a placeholder. You should edit/remove it once you have added your code. The function classify takes the gc_content as an input parameter and returns an appropriate string indicating the GC Classification. Once you have filled in the body of the classify function you should call the function from your main program in the appropriate place and use the string it returns to print out a message that matches what is expected. For example, be sure that your output for *test-moderate-gc-1.fastq* matches *test-moderate-gc-1-expected.txt* exactly using [Diff Checker](https://www.diffchecker.com/diff).\n", "\n", "After your program works for all the test files, run it on *sample_1.fastq*. Cut-and-paste just the relevant line of output from your program into __*answers.txt*__." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for working on Problem 8" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submit your work\n", "\n", "_You are almost done!_\n", "\n", "We recommend doing a quick search on this web page for answers.txt to confirm that each place we asked you to answer a question, you have answered it.\n", "\n", "At the bottom of your _answers.txt_ file, in the “Collaboration” part, state which students or other people (besides the course staff) helped you with the assignment, or that no one did.\n", "\n", "Submit the following files via this turnin page.\n", "```\n", "dna_analysis.py\n", "answers.txt\n", "```\n", "**Please** be sure to validate your dna_analysis.py program's output using the Diff Checker before submitting your assignment. You can compare your output to the files given in the expected_output directory of the homework2 files. Be sure that both the values are correct AND that the messages you are printing are formatted correctly and have no typos.\n", "\n", "Answer a **REQUIRED** survey (See [Canvas site]()) asking how much time you spent and other reflections on this assignment.\n", "\n", "_Now you are done!_" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Code cell for checking dna_analysis.py file code" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }