Homework 1: Testing and debugging difficulties

Reflect on your software development experience. Oftentimes it is lots of fun, but at times it can be frustrating.

Describe two concrete difficulties that you have encountered during software development — one about software testing and one about software debugging.

1. Think of a time when software testing took you a lot of effort. Be specific: choose a concrete incident that you remember. It should not be trivially solvable by an existing technique or tool. (Don't believe that existing tools would solve all your problems, even if they claim to do so. They might only address part of your problem, or they might have poor usability, or they might have other limitations.) It should not be merely due to your ignorance or inexperience. (But, some tools can help you more rapidly learn facts or overcome inexperience. Don't believe that all problems you have had are due to your ignorance or inexperience: they are not!)

If you cannot remember a specific incident, then put the assignment aside, think about it in the background, and come back to it later. If you still can't remember anything, you might want to look through the history of commits in projects you have worked on, or issues opened and closed, or code reviews, or TA comments.

After your name (which should be on all assignments), write 1/2 to 2/3 page discussing the problem. Give it a descriptive title (not “A testing difficulty”). Write approximately the same amount of text on each of the following (number the paragraphs in your writeup):

  1. Describe the problem in sufficient technical detail for others to understand why it was difficult. The topic sentence (that is, the first sentence) should be an informative summary or description of the problem. Discuss the underlying cause and motivate why people should care about it.
  2. Explain why the problem is not easy to solve or work around. Describe what methodologies and/or tools you used. Be specific about ways in which they were helpful and ways in which they fell short.
  3. Search for a tool or methodology that would (partially) solve your problem. In what ways is it adequate and what are its limitations?
  4. Can you invent a tool or methodology that would solve your problem? It might be similar to, or completely different from, existing ones. Describe at a high level how it would work. Think about the challenges of using such a methodology, or of building and deploying such a tool. Why do you think no one has done this already?

2. Do the same thing, but for software debugging.

3. At the end of the document:

Submit a two-page PDF file. Use a font size of 10 or 11 points, and standard margins (about 1 inch). For anything that you submit in this class, place your name and UW ID on the top of the first page.

This assignment will reward careful thought about interesting problems and issues. Please introspect deeply and thoughtfully about the challenges of developing and analyzing modern software systems. Doing so will help you in this class, and beyond.

Tips

Here is how to avoid some common problems that past students have had with this assignment.

Write your name on your homework
Be specific
The most common reason for low grades on this assignment is vagueness. Vagueness prevents the reader from understanding either the problem or the proposed solution. The assignment asks for specific events in your life, not general descriptions of an abstract problem. For example, don't merely say, “When writing tests, it is hard for me to figure out all the corner cases”, or “A common problem in C programming is segmentation faults”, or even “During my summer internship I spent a lot of time trying to read and understand ambiguous documentation.” Give enough details to understand the problem. Also give concrete specifics about how the tools or methodologies you used were useful or not.

As another example, it is not helpful or informative to say, “programmers do not use testing tools”. Such a statement needs to be clarified and supported in multiple ways.
First, what sort of testing tools is this talking about? Unit test frameworks? Mocking frameworks? Automated test generation tools? Test execution frameworks?
Second, why don't programmers use the tools, or why do they have trouble with the task? If you are not sure of the reason, then think about it, or do some research to learn the reason. At the very least, lay out some possible reasons and discuss whether you believe those are the true reasons. Even this process will increase your understanding and may lead you to come up with new and better reasons.

Discover the underlying problem
Sometimes it is easy to get tunnel vision and to get fixated on one particular approach. Think about the underlying problem you were trying to solve, in addition to the specific way you were trying to solve it. Is there a different approach to the problem that would be more effective? Try to think outside the box.
Support your claims
Don't say “Current techniques are not mature enough to do it,” without explaining why. List and briefly describe current techniques, and explain why each one falls short.
Avoid trivial problems
Avoid discussing implementation annoyances unless you can identify an underlying principle. (Example: “Windows (or Unix) lacks this feature that Unix (or Windows) has,” or “My favorite tool does not support X even though other tools do.”) Avoid mentioning difficulties in performing tasks that don't matter. (Example: “I can't determine the cyclomatic complexity of my Tcl code.”) Avoid problems that are extremely minor or that can be solved easily. (Example: “I often fail to balance delimiters before attempting to compile a file” or “There should be a public forum where people can ask and answer questions about tool X”,)

Showing example code may be helpful, but is not needed for every writeup, not even all those about issues with code.