Lecture 22: Bugs and testing

Preparation

• Read the KLEE paper. Describe one bug that KLEE cannot catch.
• (optional) mini-mc is a “minimal” implementation of KLEE’s key ideas (~15 lines of Python code). Download it and run the examples, such as mod_eqv and test_me. If you want to run it on your own machine, install Z3 first; you may use apt-get, homebrew, or build it from source code (we have already set up Z3 on attu).

Bug-finding tools

• static tools: analyze source code without running (a smart compiler)
• dynamic tools: run the code (and can try to break it)
• the line is blurred
• what bug-finding tools have you used? how do they work?
• examples
  • gcc/clang/llvm
  • sparse and some history
  • valgrind and 333 notes
  • klee
• false positives vs. false negatives

#include <stdio.h>
#include <stdlib.h>

int foo(int n)
{
	int *arr = malloc(n * sizeof(int));
	//arr[0] = 42;
	//free(arr);
	return arr[0];
}

int main(int argc, const char * argv[]) {
    printf("%d\n", foo(argc));
    return 0;
}

Testing systems code

• example: how to test your JOS kernel
  • what’s considered correct (specification)?
  • complex input sources & state transitions
  • randomly generate sytem calls?

void test_me(int x, int y) {
  int z = 2 * x;
  if (z == y) {
    if (y == x + 10) {
      crash();
    }
  }
}

• challenge: generate “useful” tests
  • what’s the chance of randomly generating (x, y) to trigger crash? 1/2⁶⁴
  • blackbox: infinite monkey theorem
• symbolic execution
  • whitebox: implementation knowledge - draw the tree of paths
  • compare the search space to the space of input
  • we need an oracle that is able to efficiently solve the path conditions

SAT/SMT solver

• termininology
  • SAT: boolean satisfiability
  • SMT: satisfiability modulo theories
• breakthrough in SAT/SMT solving
• building block for modern tools: MS Office (FlashFill), Visual Studio (IntelliTest, StaticDV), …
• try Z3 yourself; demo mini-mc

klee

• implementation
  • what state klee needs to maintain?
  • what data structure to use? OS fork good enough?
• what if the path condition is complicated?
  • example: run klee on sha1 code?
  • loops?
• how to model environment?
• false positives? false negatives?
• apply klee to JOS?