CSE 403 Assignment 2: Test coverage

In this assignment, you will download and build a piece of software, then run a tool on it. (These are very common occurrences at any job!) You will learn what a buildfile looks like. You will practice coming up to speed on codebases and tools that you are not already familiar with. You will read documentation or do web searches to solve problems that you encounter.

1. Select an open-source Java project that
   • has at least 5000 lines of non-comment, non-blank code according to cgag/loc or sloccount,
   • contains a test suite,
   • is maintained (in the previous year, has at least 10 commits and at least one issue fixed in its issue tracker), and
   • has a buildfile for Maven (pom.xml), Gradle (build.gradle), or Ant (build.xml).

   You will run various tools on this codebase during the quarter, so make sure you choose a good project!. Some "red flags" that signal projects to avoid include:

   • Most commits are authored by a single contributor.
   • The project doesn't have continuous integration enabled and passing (look for a green Travis badge!).
   • Forked projects. Use the original project instead.
   • Projects that don't follow the conventions of their build system. For example, a Gradle project whose source files aren't under src/main/java/ might be a bad choice.
   • Very large or famous projects with many moving parts. These projects often evolve their own customizations to their build scripts which make troubleshooting difficult.

   You are not tied to a project! For good project choices, everything will go smoothly. If you find yourself spending more than an hour on any one part of this assignment, consider whether switching projects would be a better use of your time!

   If, while installing sloccount, you get the error message

   install: cannot create regular file ‘/usr/local/bin/ada_count’: Permission denied

   then run sudo make install instead of make install.

2. Edit this spreadsheet to add your UW NetID and the project's repository URL (such as a GitHub URL). If your choice has already been taken, choose a different project. You may change your choice at any time, so long as you do not duplicate someone else's project. For instance, you might change your choice if it has an idiosyncratic buildfile that is difficult to integrate the coverage tool into.

   Also write the name and URL in a file README.txt.

3. Fork the project; for example, on GitHub.com, click the "Fork" button near the top right corner of the webpage. Do your work in this fork.
4. Determine a set of commands that run the project's tests and output the result into file test-execution.txt. Record these commands in file run-tests.sh. For example, your run-tests.sh file might contain this content, for a project that uses Maven as its build system:

   cd /tmp
   git clone https://github.com/user/project
   cd project
   git remote show origin | grep "Fetch URL:" >test-execution.txt
   echo SHA: $(git rev-parse HEAD) >>test-execution.txt
   time mvn test >>test-execution.txt
   

   In order to make it easier to compare a re-execution of this script, please try to minimize the number of timestamps in the output.

   (The URL and SHA are optional, but they help to show exactly what version of the software you were testing, so they help others to reproduce your results or understand any differences. You might want your run-tests.sh script to instead issue a git checkout SHA command, to ensure that others can exactly reproduce your results.)

   In file README.txt, report the number of passing and failing tests and the amount of time it took to run them.

5. Determine a set of commands that run JaCoCo, a code coverage tool, to compute coverage and output the results in file test-coverage.html or test-coverage.csv. (One way to leave the results there is to copy a file of a different name to that name.) Put the commands in file run-coverage.sh.

   JaCoCo comes with a Maven plug-in and an Ant task and can also be run from the command line. Gradle also provides a JaCoCo plugin. You should use a search engine to find the appropriate integration for the project you have chosen.

   Before you change the buildfile, check whether the project is already set up to compute coverage. For example, if the pom.xml file is already set up to run JaCoCo and you add a duplicate of those lines, you may receive very confusing error messages.

   In file README.txt, report the coverage and the time it took to compute.

6. Create a zip file that contains the 5 files above. You may write additional notes in README.txt, but this is usually not necessary.

   You may do your work anywhere you like, but running your scripts with

   bash ./run-tests.sh
   bash ./run-coverage.sh
   

   should work on a CSE HomeVM. (Note: You may need to adjust the VM's RAM to be more than 1 GB.)

7. Upload your zip file to Canvas.

At the end of the document:

• List people or resources who helped you with this assignment.
• How many hours did you spend on this assignment?

Peer Reviews

To let you see how another student did the work, and to give you the experience of following the instructions, you will review another student's work. Peer review is enabled in Canvas for this assignment.

Before you start, make sure that Java, Gradle, Ant, and Maven are installed. Also, save aside the submitted output files (test-execution.txt and test-coverage.html / test-coverage.csv), so that you can compare them to the files that result when you run the commands.

Check that the generated output matches the test and coverage results. Indicate whether all the steps worked for you, and if so, whether the generated test results and coverage numbers match the ones included. However, do not try to fix problems you encounter — if the instructions don't work, just say so (and give the exact output you received that is different than what you expected).

Advice

This assignment requires you to learn and use tools that you may not be familiar with. Start early! Doing so will save you time, because you can get help or can take a break and think about the problem. (This advice applies to any assignment where you are doing something new.)

Students in previous offering of this course have had very different experiences with this assignment. We have found that the project a student chooses is often a good predictor of success, so choose your project carefully! And remember that you can switch at any time during the assignment.

Changelog