CSE 403 Assignment 2: Test coverage

In this assignment, you will download and build a piece of software, then run a tool on it. (These are very common occurrences at any job!) You will learn what a buildfile looks like. You will practice coming up to speed on codebases and tools that you are not already familiar with. You will read documentation or do web searches to solve problems that you encounter.

  1. Select an open-source Java project that
    • has at least 5000 lines of non-comment, non-blank code according to cgag/loc or sloccount,
    • contains a test suite,
    • is maintained (in the previous year, has at least 10 commits and at least one issue fixed in its issue tracker), and
    • has a buildfile (for Maven, Gradle, Ant, or Make).
    You will run various tools on this codebase during the quarter.

    If, while installing sloccount, you get the error message

    install: cannot create regular file ‘/usr/local/bin/ada_count’: Permission denied

    then run sudo make install instead of make install.

  2. Edit this spreadsheet to add your UW NetID and the project's repository URL (such as a GitHub URL). If your choice has already been taken, choose a different project. You may change your choice at any time, so long as you do not duplicate someone else's project. For instance, you might change your choice if it has an idiosyncratic buildfile that is difficult to integrate the coverage tool into.

    Also write the name and URL in a file README.txt.

  3. Fork the project; for example, on GitHub.com, click the "Fork" button near the top right corner of the webpage. Do your work in this fork.
  4. Determine a set of commands that run the project's tests and output the result into file test-execution.txt. Record these commands in file run-tests.sh. For example, your run-tests.sh file might contain this content, for a project that uses Maven as its build system:
    cd /tmp
    git clone https://github.com/user/project
    cd project
    git remote show origin | grep "Fetch URL:" >test-execution.txt
    echo SHA: $(git rev-parse HEAD) >>test-execution.txt
    time mvn test >>test-execution.txt
    

    In order to make it easier to compare a re-execution of this script, please try to minimize the number of timestamps in the output.

    (The URL and SHA are optional, but they help to show exactly what version of the software you were testing, so they help others to reproduce your results or understand any differences. You might want your run-tests.sh script to instead issue a git checkout SHA command, to ensure that others can exactly reproduce your results.)

    In file README.txt, report the number of passing and failing tests and the amount of time it took to run them.

  5. Determine a set of commands that run a code coverage tool to compute coverage and output the results in file test-coverage.txt. (One way to leave the results there is to copy a file of a different name to that name.) Put the commands in file run-coverage.sh.

    We recommend JaCoCo, which comes with a Maven plug-in and an Ant task and can also be run from the command line. But, you may use a different tool.

    Before you change the buildfile, check whether the project is already set up to compute coverage. For example, if the pom.xml file is already set up to run JaCoCo and you add a duplicate of those lines, you may receive very confusing error messages.

    In file README.txt, report the coverage and the time it took to compute.

  6. Create a zip file that contains the 5 files above. You may write additional notes in README.txt, but this is usually not necessary.

    You may do your work anywhere you like, but running your scripts with

    bash ./run-tests.sh
    bash ./run-coverage.sh
    

    should work on a CSE HomeVM. (Note: You may need to adjust the VM's RAM to be more than 1 GB.)

  7. Upload your zip file to Canvas.

At the end of the document:

Peer Reviews

To let you see how another student did the work, and to give you the experience of following the instructions, you will review another student's work. Peer review is enabled in Canvas for this assignment.

Before you start, make sure that Java, Gradle, Ant, and Maven are installed. Also, save aside the submitted output files (test-execution.txt and test-coverage.txt), so that you can compare them to the files that result when you run the commands.

Check that the generated output matches the test and coverage results. Indicate whether all the steps worked for you, and if so, whether the generated test results and coverage numbers match the ones included. However, do not try to fix problems you encounter — if the instructions don't work, just say so (and give the exact output you received that is different than what you expected).

Advice

This assignment requires you to learn and use tools that you may not be familiar with. Start early! Doing so will save you time, because you can get help or can take a break and think about the problem. (This advice applies to any assignment where you are doing something new.)

Changelog

April 5 (for resubmission only): Made more explicit instructions about where to write commands, and not to intermix commands with other text.