Contents:
Testing increases our confidence that our code behaves correctly (i.e., meets the provided specifications). However, testing can only prove that errors do exist, not that a program is free from errors. In practice, testing is often the most practical and effective way to find bugs in your program.
Tests can be classified in various ways, according to:
Black box tests are written without knowledge of anything except for a specification.
Their advantage over clear box tests is that they are unlikely to be biased by the same misunderstandings with respect to the specification that might have caused a bug in the implementation. In many circumstances, black box tests are written by people who are not the ones who write the implementation. Also, black box tests can be re-used when the underlying implementation changes as long as the specification remains the same.
However, black box tests have the disadvantage of not being able to differentiate cases where only the implementation (and not the specification) treats inputs differently.
Clear box tests are written with full knowledge of an implementation.
Clear box tests of the specification have the advantage of being able to test that special cases in the implementation conform to the specification, even if those cases are not particularly special according to the specification alone.
The disadvantage of clear box tests is that people are likely to make the same mistakes when writing the clear box tests as they made when writing the methods that are being tested. This can result in the tests passing when the specification is not actually being accurately followed. Also, if an implementation changes, a clear box test of its specification may no longer be as effective, although it should still run correctly. (A clear box test of the implementation will have to change, of course.)
In software engineering, specifications have various different levels of visibility. For example, a client may only care that files are read and written in a specific format. This forms one level of specification. In designing the program to read and write these files, engineers should write specifications for each public method in each class they use (i.e., via Javadoc-style code comments). This forms another level of specification, which is stronger than the client's specification.
It is generally a good plan to test your code at multiple levels of specification, if they exist. In CSE 331, we request that you also divide your test cases by level of specification. Any tests on the level of the client's (the staff's) specification should be called by SpecificationTests.java. Any tests for specifications not handed out by the staff (the ones that you design) should go in ImplementationTests.java.
Here is a good rule of thumb: If your test could compile and run as a valid test against any other student's code, then it should go in SpecificationTests.java. Otherwise, it should go in ImplementationTests.java.
As a brief aside, notice that black-box testing and glass-box testing DO NOT correspond to the SpecificationTest and ImplementationTest suites that we have you build in CSE 331. Instead, our rule of thumb is that specification tests are tests that should pass on everybody's code base, while implementation tests are tests that may only work on your code. Thus, the implementation tests and specification tests are completely orthogonal to the concepts of black-box and glass-box testing.
Q:Why would you build a black-box implementation test?
A:Because you will want to test your internal specifications. In PS3 and beyond you are given the freedom to write some of your own class specifications, including defining their public interfaces. Therefore, any black-box tests you write based on your specifications will only work on classes you write, and so by the rule of thumb given above those tests should go in the ImplementationTests suite.
Unit tests are tests for one method of one class. They test your system in a fine-grained way, to help you be sure that you're building your system out of solid building blocks.
Integration tests are tests of how the system fits together. They usually involve more complicated operations than unit tests, and are meant to tests how the system will work in practice. An example is the practice of reading in a file that defines some operations, performing those operations, writing out a result file, and comparing the result file to a version (called the golden file) that has been hand-inspected and verified to be correct.
Tests are written in a separate class from the code being tested — this separates distinct concerns, and it keeps the tests from cluttering the code. However, this makes it impossible to directly test private methods, which are not accessible outside the class in which they appear. (When the tests are in a separate package than the code being tested, a similar problem occurs for methods with protected or default accessibility.)
To test private methods, we recommend that you write the tests in the same class as the method itself, in a method with greater visibility. For instance, to test private method foo, you would write a public method named testFoo. The specification of testFoo is the same as that of checkRep (another public method that tests private, inaccessible details of the class). Each of these takes no arguments and return no results, but throws an exception if any of its tests fail.
Other approaches are less desirable. Making the method public breaks modularity, and clients could come to depend on it, restricting the flexibility of the implementer. Using reflection to temporarily make the method public is ugly and hard to understand. Not testing the method is even worse, as bugs in a helper method may be much harder to debug when they are exposed only by incorrect behavior of a method that calls them.