Testing and Debugging
The scientific process of debugging, the information problem, and the value in automated testing.
Kevin Lin, with thanks to many others.
1
Ask questions anonymously on Piazza. Look for the pinned Lecture Questions thread.
Feedback from the Reading Quiz
2
The reading discussed debugging practices with an emphasis on gathering information.
Real world analogues
Rubber duck assisting with debugging (Tom Morris/Wikimedia)
3
Minimal working example
Rubber duck debugging
Scientific method
Experienced programmers can sometimes seem to solve problems almost intuitively because of how much practice they've had getting unstuck. While they get stuck just as often as you or me, they’ve exercised their problem-solving muscle enough to have an idea of which debugging strategy to try next. This learning doesn’t come immediately: it takes a lot of practice to develop these metacognitive skills.
ArrayQueue Demo
4
The Role of Information
5
What does debugging a program look like? (Julia Evans); The Debugging Mindset (Devon H. O’Dell/ACM Queue)
How are bugs fixed? Here’s one proposal.
Productive changes fix bugs.
Information gathered about the system informs productive changes.
A hypothesis guides information gathering and testing.
Things we know about the problem inform how we choose hypotheses.
ArrayQueue maintains certain invariants.
Unexpected result after add and remove.
A bug exists in the ArrayQueue isEmpty method.
Information ???
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
1
2
3
?: Do we have reason to question the things we know about the problem? Or the hypothesis?
Generating Hypotheses
6
A good hypothesis identifies the cause of failure separately from where and when the program actually fails. The state of the ArrayQueue determines the behavior of isEmpty.
Item[] data
int size
int front
int back
State
A bug exists in the ArrayQueue isEmpty method.
Information ???
2
3
The Debugging Mindset (Devon H. O’Dell/ACM Queue)
A good hypothesis describes a problem and is both testable and falsifiable. While this hypothesis is both, it doesn’t really yield much more information about the problem or how to fix it.
Descriptive Hypotheses
7
A good hypothesis identifies the cause of failure separately from where and when the program actually fails. The state of the ArrayQueue determines the behavior of isEmpty.
The hypothesis on the left suggests more about the problem than the one on the right.
A bug exists in the ArrayQueue isEmpty method.
The Debugging Mindset (Devon H. O’Dell/ACM Queue)
The size variable is not set correctly, causing isEmpty to return false.
Q1: Which of the hypotheses below do you think will yield more information upon answering?
8
Tests as a Source of Information
ArrayQueue1<Integer> queue = new ArrayQueue1<>();
queue.add(1);
queue.remove();
queue.add(3);
queue.remove();
queue.remove();
queue.add(6);
queue.remove();
System.out.println( "isEmpty() expected true, got " + queue.isEmpty());
?: What does this test check?
Propose a new hypothesis that, upon answering, could yield more information.
9
Gathering Information
10
Debugging is about integrating various different sources of information. Let’s try out a few methods of gathering information to understand the bug.
Trying new inputs.
Writing a unit test to reproduce the bug.
Explaining to yourself the behavior of each line of code.
Searching online to understand what error messages mean.
Changing or removing code.
Poking at memory values with a debugger or print statements.
?: Bugs often appear away from their root causes. How does each information gathering method help us learn more about the problem?
Ad-Hoc Testing
queue.add(1);
queue.remove();
queue.add(3);
queue.remove();
queue.remove();
queue.add(6);
queue.remove();
System.out.println( "..." + queue.isEmpty());
11
queue.add(0);
queue.remove();
queue.add(2);
queue.add(3);
queue.add(5);
queue.add(6);
System.out.println( "..." + queue.remove());
ArrayQueue1.main
ArrayQueue2.main
JUnit Testing
12
Testing is a means of gathering information. But because unit tests are just programs, we can automate it and continuously gather information.
Simple JUnit Testing
Call org.junit.Assert.assertEquals(...) to check that the expected equals the actual. If not, program terminates with a verbose message.
We can use this in place of writing out long print messages just to compare two arguments.
JUnit supports many more methods. (Check the online JUnit documentation for more.)
assertEquals
assertFalse
assertNotNull
13
org.junit.Assert.assertEquals(expected, actual);
ArrayQueueTest method
Better JUnit Testing
The messages output by JUnit are kind of ugly, and invoking each test manually is annoying.
IntelliJ has built-in support for JUnit.
Annotate each test (Java method) with @org.junit.Test.
Change all test methods to non-static.
Use IntelliJ’s built-in JUnit runner to run all tests and tabulate results.
This is called boilerplate code. IntelliJ can generate this code for you!
14
@org.junit.Test
ArrayQueueTest
To have IntelliJ generate the boilerplate code in a class ArrayQueue Test,
With the cursor placed somewhere inside the class’s curly braces, press Alt+Insert (on Windows/Linux) or Command+N (on Mac) to open the options for generating code.
In the pop-up menu, select “Test Method” and then choose the JUnit test option.
Fill in a name for the test.
Write the body of the test.
Protip: You can also generate all sorts of other boilerplate from this keyboard shortcut, such as “public static void main(String[] args)”.
Even Better JUnit Testing
Don’t want to type out the name of the library (org.junit.Test, org.junit.Assert.assertEquals)?
To workaround this annoyance, start every file with two import statements.
15
import org.junit.Test;
import static org.junit.Assert.*;
ArrayQueueTest
The Role of Information
16
What does debugging a program look like? (Julia Evans); The Debugging Mindset (Devon H. O’Dell/ACM Queue)
How are bugs fixed? Here’s one proposal.
Productive changes fix bugs.
Information gathered about the system informs productive changes.
A hypothesis guides information gathering and testing.
Things we know about the problem inform how we choose hypotheses.
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
Two new regression tests
They’re called regression tests because we want to ensure that future changes don’t break these test cases that we know our program has had difficulty with in the past.
Testing as Planning
Suppose we’re implementing ArrayDeque from HW 2.
Describe a unit test we might want to write for ArrayDeque.
What behaviors does this test check? Describe in terms of the methods it checks as well as concepts like contracts, invariants, etc. that we’ve discussed in class.
17
Q
Implementer
Client
ADT
?: How did the ad-hoc tests for ArrayQueue1 and ArrayQueue2 expose particular bugs? What was special about those tests?
Q1: Describe a unit test we might want to write for ArrayDeque. Recall that the Deque interface expects methods such as addFirst, addLast, removeFirst, removeLast, and get.
Q2: What behaviors does this test check? Describe in terms of the methods it checks as well as concepts like contracts, invariants, etc. that we’ve discussed in class.
Describe a unit test we might want to write for ArrayDeque.
18
Testing as Planning
Not only does running a test improve our understanding of a problem, but so does writing a test!
Tests are hard to write, but easy to run.
Maximize the benefit of testing by writing tests first (or early) and code afterwards.
19
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
What does debugging a program look like? (Julia Evans)
“I’m almost done, I just need to make sure it works.”
– Famous last words
If testing is left until after all of the code is written, we lose any opportunities to gather information along the way and fix bugs as they come up! We may even be solving the wrong problem altogether.
?: How does testing serve as a form of planning?
?: What makes writing good tests so challenging?
Test-Driven Development
Identify a new feature.
Write a unit test for that feature.
RED: Run the test. It should fail.
GREEN: Write code that passes test.
REFACTOR: Improve code quality.
20
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
What does debugging a program look like? (Julia Evans); Red-Shirt, Red, Green, Refactor - A TDD Fairytale (Ryan Tablada)
?: Testing is just one tool in the information gathering toolbox. We’ve seen how testing can be a forming of planning. How do other information gathering methods inform our planning processes?
Running tests is virtually free compared to other sciences
21
Algorithms (Robert Sedgewick, Kevin Wayne/Princeton)
Chemistry(~1 test/day)
Biology(~1 test/month)
Physics(~1 test/month+)
CS(1+ test/sec)
You can even automate your testing workflow so that every change to the code is automatically verified against the entire suite of unit tests to prevent regressions. Search for “Toggle auto-test”.
The Role of Information
22
What does debugging a program look like? (Julia Evans); The Debugging Mindset (Devon H. O’Dell/ACM Queue)
How are bugs fixed? Here’s one proposal.
Productive changes fix bugs.
Information gathered about the system informs productive changes.
A hypothesis guides information gathering and testing.
Things we know about the problem inform how we choose hypotheses.
ArrayQueue maintains certain invariants.
Unexpected result after add and remove.
The remove method decrements the size variable even when the queue is empty.
Modify the remove method to handle the special case of removing if empty.
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
1
2
3
?: What are the differences between this new hypothesis and the hypothesis that we started with? How did we get from the starting hypothesis to this new hypothesis?