CSE 490E1 Assignment 4: Test generation

Due: Tuesday, January 17, 2017 at 11:59pm, via Canvas

The goal of this assignment is to appreciate the difficulty of writing tests, and the strengths and weaknesses of automatic systems for doing so.

You will submit this assignment as a text file (.txt). Clearly indicate each question in your .txt file. Ensure that each answer gives enough information for someone else to reproduce your results and conclusions.

  1. Fork the open-source Java project that you used in assignment 2. You may choose another one that satisfies the same criteria, if you have trouble with the one you used for assignment 2. Give the URL for your fork.
  2. Find some aspect of the project's behavior that is not tested. Briefly explain how you know it is not tested. Be specific; for example, don't just say "the code wasn't covered", but give the exact commands to produce coverage information and say where to look in the output.
  3. Write one or more tests that improve the quality of the test suite. These could be unit tests, end-to-end tests, or something else. Give a URL to a commit that adds the tests. Give a command that computes the coverage, and the difference in coverage before and after adding your tests.
    Most likely, you write a test that increases coverage. It would be even more useful to write a test that reveals a defect. Briefly speculate about how would you do so and how difficult it would be. As a stretch goal after completing the rest of the assignment, you could try to do so.
  4. Run the Randoop test generation tool on the project. This will create two test suites: one that reveals errors that are already in the program, and one that creates regression tests. Give the commands that you used to run Randoop; most likely, this involves modifications to the build files.
  5. How many error-revealing tests were generated? If any were generated, then choose one, fix the underlying bug, and add a regression test to the project's test suite. (You probably want to simplify Randoop's generated test, which may be unduly complicated.) Show the Randoop test and give a URL to the commit that fixes the bug and adds a test.
  6. How many regression tests were generated? (The regression tests are very complicated; don't spend too much time trying to read them.) Point out some code that that the original test suite covers but the Randoop test suite does not cover. Why wasn't Randoop able to cover it? Point out some code that the Randoop test suite covers but the original test suite does not. Why didn't the developers write tests to cover that code?
  7. In a few paragraphs, comment on the utility of the Randoop tool. What worked well? What worked poorly? When would you want to use it (if ever!)? How do you think it should be improved?

Upload your text file with results to Canvas.