Software Experimentation
The scientific process of debugging, the information problem, and designing experiments for correctness and efficiency.
Kevin Lin, with thanks to many others.
1

ArrayQueue1
2

Software Experiments
3
What does debugging a program look like? (Julia Evans); The Debugging Mindset (Devon H. O’Dell/ACM Queue)
How are bugs fixed? Here’s one proposal.
Productive changes fix bugs.
Information gathered about the system informs productive changes.
A hypothesis guides information gathering and testing.
Things we know about the problem inform how we choose hypotheses.
ArrayQueue maintains certain invariants.
Unexpected result after add and remove.
A bug exists in the ArrayQueue isEmpty method.
Information ???
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
1
2
3

Generating Hypotheses
4
A good hypothesis identifies the cause of failure separate from where and when the program actually fails. The state of the ArrayQueue determines the behavior of isEmpty.
Item[] data
int size
int front
int back
State
A bug exists in the ArrayQueue isEmpty method.
Information ???
2
3
The Debugging Mindset (Devon H. O’Dell/ACM Queue)

Generating Hypotheses
5
A good hypothesis identifies the cause of failure separate from where and when the program actually fails. The state of the ArrayQueue determines the behavior of isEmpty.
The hypothesis on the left suggests more about the problem than the one on the right.
A bug exists in the ArrayQueue isEmpty method.
The Debugging Mindset (Devon H. O’Dell/ACM Queue)
The size variable is not set correctly, causing isEmpty to return false.
?: What is it about the hypothesis on the left that suggests more about the problem?

6
Tests as a Source of Information
ArrayQueue1<String> queue = new ArrayQueue1<>();
queue.add("a");
queue.remove();
queue.add("big");
queue.remove();
queue.remove();
queue.add("car");
queue.remove();
System.out.println(    "isEmpty() expected true, got " + queue.isEmpty());
A good hypothesis describes a problem and is both testable and falsifiable.

Q1: Propose a new hypothesis from the test code.

Propose a new hypothesis from the test code.
7
A

Gathering information
8
Debugging is about integrating various different sources of information to identify the source of an error.

Trying new inputs.
Writing a unit test to reproduce the bug.
Explaining to yourself the behavior of each line of code.
Searching online to understand what error messages mean.
Changing or removing code.
Poking at memory values with a debugger or print statements.

?: Bugs often appear away from their root causes. How does each information gathering method help us learn more about the problem?

Software Experiments
9
What does debugging a program look like? (Julia Evans); The Debugging Mindset (Devon H. O’Dell/ACM Queue)
How are bugs fixed? Here’s one proposal.
Productive changes fix bugs.
Information gathered about the system informs productive changes.
A hypothesis guides information gathering and testing.
Things we know about the problem inform how we choose hypotheses.
ArrayQueue maintains certain invariants.
Unexpected result after add and remove.
The remove method decrements the size variable even when the queue is empty.
Modify the remove method to handle the special case of removing if empty.
The point here is that information is the most important thing and you need to do whatever’s necessary to get information.
1
2
3
?: What are the differences between this new hypothesis and the hypothesis that we started with? How did we get from the starting hypothesis to this new hypothesis?

ArrayQueue2
10

Storytelling with Representation Invariants
Give an invariant for ArrayQueue2.front.
11
How did the program break the invariant?
A

Ad-Hoc Testing
queue.add("a");
queue.remove();
queue.add("big");
queue.remove();
queue.remove();
queue.add("car");
queue.remove();
System.out.println(    "..." + queue.isEmpty());
12
queue.add("a");
queue.remove();
queue.add("big");
queue.add("car");
queue.add("doesn't");
queue.add("eat");

System.out.println(    "..." + queue.remove());
ArrayQueue1.main
ArrayQueue2.main

JUnit tests
13
Testing is a tool for gathering information about the correctness and efficiency of programs. We can create regression tests to ensure that future changes don’t break these cases again as we make changes to our program.

Running tests is virtually free compared to other sciences
14
Algorithms (Robert Sedgewick, Kevin Wayne/Princeton)
Chemistry(~1 test/day)
Biology(~1 test/month)
Physics(~1 test/month+)
CS(1+ test/sec)

Experimentation for correctness and efficiency
15
While testing software is not a major focus of this course, it’s an important tool for analysis of algorithms.

There are three issues with using random testing.

Reproducibility. Seed the random number generator to get the same sequence of “random” numbers each time.
Reference model. Random tests often need a reference implementation to compare for correctness. It’s not always easy to come up with a reference.
Debuggability. Testing infrastructure is needed to aid in interpreting results.

Once we have a correct implementation, we can use the Stopwatch class to compute the time between runs.