CSE 403 Topic: Testing

A test is an input to some piece of software and an associated expected output. The test is said to pass if the software produces the expected output when passed the input. Otherwise it is said to fail.

"Testing shows the presence, not the absence of bugs"

Edsger Dijkstra

Wikipedia has a decent (if extremely long) article on different kinds of software testing: https://en.wikipedia.org/wiki/Software_testing

Here are some important kinds of testing:

• regression testing: tests that are run after each change to the software to make sure that no old behavior changed unexpectedly
• random testing/fuzzing: determining expected outputs is challenging, but finding inputs isn't. If you know some behaviors (like crashes or segfaults) that are bad, you can use random inputs to try to get the software to exhibit them.
• unit testing: simple tests that check that every small piece of the software does what you want it to - often at the method or procedure level. Usually part of a regression test suite.
• mutation testing: how do you know if your tests are good enough? Mutation testing is inserting intentional error into your programs and seeing if your tests can catch them.

Interesting testing tools and frameworks:

• JUnit: an extremely common Java unit testing framework.
• American Fuzzy Lop (AFL): a fuzzer.
• Randoop: a random test input generator for Java.
• OSS-Fuzz: Google constantly fuzzes several important open-source libraries, looking for security bugs.
• PIT: Java mutation testing.
• JaCoCo: test coverage for Java.

Interesting resources:

• Academic review of mutation testing research: http://crest.cs.ucl.ac.uk/fileadmin/crest/sebasepaper/JiaH10.pdf
• Blog post on the value of test coverage: https://martinfowler.com/bliki/TestCoverage.html
• Academic study on whether code coverage actually correlates with test suite effectiveness: https://dl.acm.org/citation.cfm?id=2568271
• Old but excellent blog post on the importance of unit testing: https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/

Project ideas:

• write a new (random, unit, mutation) testing tool for your favorite language - especially if it doesn't have one!
• coverage tools are aging but still widely used. Could you write a better one?
• can you think of a clever way to automatically generate tests for some kind of software you've written before?