CSE143 Notes 4/21/06

Complexity & Efficiency

Question: how fast is this method?

   public double sum(double[] data) {
     double result = 0.0;
     for (int k = 0; k < data.length; k++) {
       result = result + data[k];
     }
     return result;
   }

What is the question asking? How many microseconds? Compared to what?

Goal: We'd like some way to compare the performance of two algorithms that do the same task or two different data structures that implement the same abstraction using different algorithms. We want this to be independent of any particular implementation or machine. It turns out that the same ideas and techniques work for comparing execution time, space, and other resource usage. We'll focus on execution time, since this is usually the one we're most interested in for the kinds of problems we've been looking at.

To analyze an algorithm, we need to do the following:

1. Describe the size of the problem being solved. This is typically something like the size of the data structure being processed or the magnitude of some parameter, like the number of steps to run a simulation.
2. Count the number of steps needed to solve the problem as a function of the problem size.

What's a step? The idea is that we want to think abstractly about the elementary operations that a simple computing machine can perform. As a first approximation, a step is a simple operation or statement in a programming language like Java. Examples:

• A single arithmetic operation (+, -, *, /, %)
• Assignment of simple primitive types or of references to data objects, but not making a copy of an entire object
• Array subscripting
• Simple conditional tests (<, <=, ==, &&, ||), but not potentially complex operations like .equals or .compareTo
• Initializing parameters with argument values in a method call, but not the cost of evaluating the argument expressions if these are complex
• The basic "jump over and back" operations in method call and return.

The costs of more complex operations are typically the sum of the costs of their components. For example:

• Sequence of statements: s1; s2; ...; sn. Sum of the costs of the individual statements
• Conditional: if (cond) s1 else s2. Typically we're interested in how much time could potentially be needed, so we use the cost of evaluating the condition plus the maximum cost of either s1 or s2. There are times where the average cost is more useful than the worst-case cost, but that analysis can be more complex, involving the expected probability of executing the different statements s1 and s2.
• Loops: while (cond) s. Total cost is the product of the number of iterations times the cost of the statements executed during each iteration. This extends to nested loops. If s itself is a loop, the total cost is the number of times the outer loop is executed times the cost of executing the inner loop each time around the outer one.
• Method calls. The sum of the cost of evaluating the arguments (constant for simple things, not constant for more complex things) plus the cost of passing the parameters (typically constant for Java code) plus the cost of jumping over and back (typically constant) plus the cost of executing the method body.

Remember that all of these costs should be measured relative to the problem size. Some of the costs of an algorithm, even fairly large ones, won't depend on the problem size, others will.

Once we've done this analysis, we will wind up with a number that says, for instance, it takes 25n + 3n² + 17 steps to solve a problem of size n. Another algorithm might take 42n + 6(n log n) + 300 steps to solve the same problem. Now, which is better?

Certainly for small values of n, the first algorithm requires fewer steps. But in general we're interested in how an algorithm behaves for large problems - after all, small problems can almost always be solved so quickly that it really doesn't matter. So what we're interested in is finding the asymptotic complexity of an algorithm - its cost as the problem size gets large. For this sort of analysis, only the high-order terms really matter.

Rule of thumb: to compare the asymptotic complexity of two algorithms, drop all but the high-order terms and ignore the constants. So, for the examples above, what matters about 25n + 3n² + 17 is that it's proportional to n², and 42n + 6(n log n) + 300 is proportional to n log n. For large values of n, then, the second one is faster.

There is a standard notation in computer science that captures this idea: Big-O notation. Definition: If f(n) and g(n) are two complexity functions, we say that f(n) = O(g(n)) [pronounced "f(n) is order g(n)"] if there is some constant c such that f(n) ≤ c g(n) for all sufficiently large n.

Exercise: give an informal proof that 5n+3 is O(n)

Exercise: give an informal proof that 5n² + 42n + 17 is O(n²)

Fine print:

• The notation f(n) = O(g(n)) is not an equality. It is shorthand for "f(n) grows at most like g(n)" or "f grows no faster than g" or "f is bounded above by g"
• O( ) notation normally is a worst-case analysis. It's generally useful in practice, but there are times where an average-case or expected-time analysis is more useful.

Complexity classes. There are several common, basic complexity classes. You should know these in order, and be able to draw graphs of them..

• Constant: O(k) or O(1)
• Logrithmic: O(log n)
• Linear: O(n)
• "n log n": O(n log n)
• Quadratic: O(n²)
• Cubic: O(n³)
• ...
• Exponential: O(2ⁿ)

Times that are O(nk) or better are called polynomial time. Algorithms that run in polynomial time are generally considered to be feasible; algorithms that require exponential time are not.