CSE 403 Assignment 3: Test generation

The goal of this assignment is to appreciate the difficulty of writing tests, and the strengths and weaknesses of automatic systems for doing so.

You will submit this assignment as a zip file (.zip) containing two files, README.txt and commands.sh.

  1. Re-use the fork of the open-source Java project that you used in assignment 2. You may choose another project that satisfies the same criteria, if you have trouble with the one you used for assignment 2. Give the URL for your fork.
  2. Find some aspect of the project's behavior that is not tested. Briefly explain how you know it is not tested. Be specific. For example, don't just say "the code wasn't covered". Instead, give a file to examine that was created by the commands in commands.sh, say where to look in the output, and how to interpret the information there.
  3. Manually write one or more tests that improve the quality of the test suite. These could be unit tests, end-to-end tests, or something else. Give a URL to a commit that adds the tests. Point at files (created by your commands.sh file) that contain the coverage before and after the change. Summarize the difference in coverage before and after adding your tests.

    If the coverage report gives a truncated result (say, an integer coverage percentage), then your new tests do not need to improve the the truncated result (you don't need to improve the whole test suite's coverage by a full percentage point). Change the coverage tool's output to include more digits of precision, or indicate a report that shows that your addition really did improve coverage.

  4. Run the Randoop test generation tool on all the classes of the project. This will create two test suites: one that reveals errors that are already in the program, and one that creates regression tests. Commit any needed changes to the project's build files.

    The commands.sh should include commands that download and install Randoop before running it.

    If you do not know what the Java classpath is, please spend a moment to learn about it. Briefly, it tells the JVM where to find classes. When you run Randoop, the JVM will need to load and run classes from Randoop itself, and also classes from the application under test (and any libraries it uses). Be sure that the classpath contains both.

    You can typically find a .jar file in a subdirectory build/libs/ in the project after you build it. If that is a "fat jar" that also contains all the libraries that the project uses, then it is enough. If that is a "skinny jar" that contains only the project's classfiles, but not those of libraries it uses, then you need to either make a fat jar (use the Maven shade plugin or the Gradle shodow plugin) or include all the libraries on the classpath.

    If Randoop crashes due to a bug (this is usually indicated by "Randoop failed in an unexpected way." and a stack trace), submit a bug report and skip the two questions about the tests that it produces.

    If you are having trouble running Randoop, first make sure that you can run this command from its user manual:

    java -Xmx3000m -classpath $RANDOOP_JAR randoop.main.Main gentests --testclass=java.util.TreeSet --time-limit=10

    The invocation on your project will be rather different; this invocation is just to make sure you installed Randoop correctly. If you didn't set environment variable RANDOOP_JAR, then use the path directly in place of $RANDOOP_JAR, which is how bash refers to a variable.

  5. How many error-revealing tests were generated? If there are any, choose the first one, and determine whether it really reveals an error, or it is a false positive. If it is a false positive, explain why and start over examining the next one. If the first three are all false positives, you can stop.

    For the first test that is not a false positive, fix the underlying bug and add a regression test to the project's test suite. (Before you try to understand the regression test, minimize it, or run Randoop in the first place with the --minimize-error-test command-line option.) Show the (minimized) Randoop test and give a URL to the commit that fixes the bug and adds a test.

  6. How many regression tests were generated? (The regression tests are very complicated; don't spend too much time trying to read them.) Point out some code that the original test suite covers but the Randoop test suite does not cover. Why wasn't Randoop able to cover it? Point out some code that the Randoop test suite covers but the original test suite does not. Why didn't the developers write tests to cover that code?
  7. In 1-2 paragraphs, comment on the utility of the Randoop tool. What worked well? What worked poorly? When would you want to use it (if ever!)? How do you think it should be improved?

At the end of the document:

Upload your zip file to Canvas.

Peer Reviews

This assignment will be peer reviewed. There is a rubric on canvas which has a set of Yes/No questions. If your answer is "No" for any of the criteria, give a short description of what failed for you. Also give comments on whether the given documentation was sufficient and easy to follow.

Peer review rubric

Common mistakes to avoid


Wed April 11, 11:00am
Clarified point #3: You must manually write the tests, and you don't have to increase overall coverage by a whole 1%.
Fri April 13, 8:00am
Changed point #4: Put commands in a separate file commands.sh rather than in your main README.txt file.
Clarified point #4: If Randoop crashes, submit a bug report.
Clarified point #5: If Randoop generates no error-revealing tests, just report that fact.
Mon April 16, 3:00pm
Clarified that submission is a .zip file, not a .txt file.
Added text explaining that your script should clone your repo and it should not be included in your .zip file.
Mon May 7, 10:00am
Copied peer review rubric over from Canavs.