The number of steps involved will depend on how many times you have to cut the set of possible answers in half. If you start with n answers, eventually you will get down to just one. The number of steps, then, can be computed as:
n / 2 / 2 / 2 ... / 2 = 1
n / 2^? = 1
n = 2^?
? = log2(n)
I contrasted this with the way that we wrote the indexOf method for both the
ArrayIntList and the LinkedIntList. In both cases we iterated through the list
from beginning to end until we either found the value or ran out of values.
This is an approach known as linear search.I said that clearly the binary search is faster than the linear search, but how much faster? The linear algorithm exams all n values potentially. If a value is not found, it has to examine all of them. Even if it is found, it will tend to look at half of them on average if the value appears once in the list. So we are comparing an algorithm that takes n steps versus and algorithm that takes log2(n) steps. That is a huge difference. I mentioned that an appoximation I often use to reason about this is that 2^10 is about equal to 10^3 (the actual values are 1024 versus 1000).
Using this approximation, we can say that for one thousand values to search, we are talking about an algorithm that takes 10 steps versus an algorithm that takes one thousand steps. For a million values, it's 20 steps versus a million. For a billion values, it's 30 steps versus a billion. For a trillion it is 40 steps versus a trillion. And so on.
Computer scientists characterize differences like these by talking about the complexity of an algorithm.
The word "complexity" can be interpreted in many ways. It sounds like a measure of how complex or how complicated a program is. For example, jGRASP has a tool under the File menu that allows you to create a "Complexity Profile Graph" of your code, which is a software engineering concept that is somewhat similar to this. But that's not how computer scientists use the term most often. When we refer to the complexity of an algorithm or a code fragment, we most often are referring to the resources that it requires to execute. The two resources that we are generally most interested in are:
Of these two, the resource that computer scientists most often refer to when talking about complexity is time. In particular, we are interested in the growth rate as the input size increases. We begin by deciding on some way to measure the size of the input (e.g., the number of names to sort, the number of numbers to examine, etc) and call this "n". We are interested in what happens when we change n. For example, if it takes time "t" to execute for n items, how much time does it take to execute for 2n items?
I pointed out that this is one of the few places where computer science is actually like a science. Some instructors ask their students to collect empirical timing data for different input sizes and have them plot these values to see if the plot matches the prediction. Unfortunately, these experiments are more difficult to perform on modern computers because features like cache memory skew the results. The important thing is that the predictions hold for large values of n.
I then mentioned a simple rule of thumb that you can apply to Java programs to figure out the complexity of a code segment. I mentioned that it "almost" works. The idea is to find the line of code that is executed most often. In thinking about this, you have to be careful how you count. For example, with a for loop, we'd count the loop itself as executing just once, but the statements controlled by the loop might be executed many times. Of course, a for loop can be inside a for loop in which case the inner loop is executed multiple times. But think in terms of how many times you enter the loop when counting the number of executions of the line of code that begins with "for". You also have to consider how many times various lines of code get executed that you didn't write. There is a method called Arrays.sort that will sort an array and you can't count it as one line of code being executed when you call it. You would have to count how many times its lines of code are executed as well. Programmers use tools known as profilers to give them data about how many times each line of code is executed.
I pointed out that I see a lot of undergraduates who obsess about efficiency and I think that in general it's a waste of their time. Many computer scientists have commented that premature optimization is counterproductive. Don Knuth has said that "Premature optimization is the root of all evil." The key is to focus your attention where you can get real benefit. The analogy I'd make is that if you found that a jet airplane weighed too much, you might decide to put the pilot on a diet. While it's true that in some sense every little bit helps, you're not going to make much progress by trying to slim down the pilot when the plane and its passengers and cargo weigh so much more than the pilot. I see lots of undergraduates putting the pilot on a diet while ignoring much more important details.
In terms of the growth rate of different algorithms, I mentioned that some of your intuitions from calculus will be helpful. You've probably been asked to solve problems like figuring out what the limit is as you approach infinity of an expression like this:
n^3 - 18 n^2 + 385 n + 708
--------------------------
0.005 n^4 - 13 n^2 + 73842
When you solve a limit like this, you ignore things like coefficients and you
ignore small terms. What matters here is that you basically have:
n^3
---
n^4
The rest is noise. So this is something that you know is going to approach 0
because eventually the n^4 will dominate the n^3 no matter what the
coefficients and lower-order terms are. We use similar reasoning with
complexity. We ignore constant multipliers and we ignore lower order terms to
focus on the main term.I said that different algorithms naturally fall into different complexity classes:
The following table presents several hypothetical algorithm runtimes as an input size N grows, assuming that each algorithm required 100ms to process 100 elements. Notice that even if they all start at the same runtime for a small input size, the ones in higher complexity classes become so slow as to be impractical.Then I talked about a specific problem. The idea is that we have a list of integers, both positive and negative, and we want to find the subsequence that has the highest sum. If there weren't any negative integers, you'd always include all of the numbers. But because some of them can be negative, it might be the case that some portion of the list has a sum that is greater than any other sequence from the list. The subsequences always involve taking a contiguous chunk of the list. This particular problem has often been used by Microsoft as an interview question, probably because there are different ways to solve it, some of which are much faster than others.
input size (N) O(1) O(log N) O(N) O(N log N) O(N2) O(N3) O(2N) 100 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 200 100 ms 115 ms 200 ms 240 ms 400 ms 800 ms 32.7 sec 400 100 ms 130 ms 400 ms 550 ms 1.6 sec 6.4 sec 12.4 days 800 100 ms 145 ms 800 ms 1.2 sec 6.4 sec 51.2 sec 36.5 million years 1600 100 ms 160 ms 1.6 sec 2.7 sec 25.6 sec 6 min 49.6 sec 42.1 * 1024 years 3200 100 ms 175 ms 3.2 sec 6 sec 1 min 42.4 sec 54 min 36 sec 5.6 * 1061 years
As an example, suppose you have an array that stores these values:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
The maximum sum is obtained by adding up the values from index [3] through
[9]:
4 + 6 + 10 + -18 + 5 + 5 + 11 = 23
It might seem odd that you include -18, but it's because including that allows
you to include the three numbers that come before, which add up to 20 (a net
gain of 2 for the overall sum).There is a simple way to solve this that involves picking each possible subsequence. We can have one for loop that generates each possible starting point and another for loop that generates each possible stopping point:
for (int start = 0; start < list.length; start++) {
for (int stop = start; stop < list.length; stop++) {
look at the numbers from start to stop
}
}
So how do we "look at the numbers from start to stop"? We can write a loop
that adds up each of those numbers:
int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i];
}
And once we have that sum, we can compare it against the maximum sum we've seen
so far and reset the maximum if this sum is better:
if (sum > max) {
max = sum;
}
Putting these pieces together and including some initialization outside the
loop, we end up with the following code:
int max = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++)
for (int stop = start; stop < list.length; stop++) {
int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i];
}
if (sum > max) {
max = sum;
maxStart = start;
maxStop = stop;
}
}
That's the first approach. The line that is executed most often in this
approach is the "sum += ..." line inside the innermost for loop (the "i" loop
that adds up the list). It is nested inside three different loops, each of
which executes on the order of n times. So we would predict that this code
would be an O(n3) algorithm.Then I asked how the algorithm could be improved. How can we do this faster? The bottleneck is the line that is adding up individual numbers and the key to improving the algorithm is noticing how we're doing a lot of duplicate work. Think about what happens the first time through the outer loop when "start" is equal to 0. We go through the inner loop for all possible values of "stop". So suppose the list is 2000 long. We're going to compute:
the sum from 0 to 0
the sum from 0 to 1
the sum from 0 to 2
the sum from 0 to 3
the sum from 0 to 4
...
the sum from 0 to 1999
Those are all the possibilities that start with 0. We have to explore each of
these possibilities, but think about how we're computing the sums. We have an
inner "i" loop that is computing the sum from scratch each time. For example,
suppose that we just finished computing the sum from 0 to 1000. We next
compute the sum from 0 to 1001. But we start from the very beginning and have
i go through all of the values 0 through 1001 when we've just computed the sum
from 0 to 1000. That was a lot of work. Then you throw away that sum and
start from scratch to add up the values from 0 to 1001. But why start back at
the beginning? If you know what the values from 0 to 1000 add up to, then just
add the value at position 1001 to get the sum from 0 to 1001.So the key is to eliminate the inner "i" loop by keeping a running sum. This requires us to move the initialization of sum from the inner loop to the outer loop so that we don't forget the work we've already done.
int max = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++) {
int sum = 0;
for (int stop = start; stop < list.length; stop++) {
sum += list[stop];
if (sum > max) {
max = sum;
maxStart = start;
maxStop = stop;
}
}
}
In this code the most frequently executed statements are inside the for loop
for "stop" (the line that begins "sum +=" and the if). There are other lines
of code that you could argue also tie with this, like the for loop test and for
loop increment. Each of these lines of code is inside an outer loop that
iterates n times and an inner loop that iterates an average of n/2 times. So
this is an O(n2) algorithm.I mentioned that there is a third algorithm, although I wouldn't have time to discuss it in detail and it is the most difficult to understand, so I did not attempt to prove its correctness. I did, however, try to explain the basic idea. The key is to avoid computing all of the sums. We want to have some heuristic that would allow us to ignore certain possibilities. We do that with a single loop that computes for each value of i the highest possible sum you can form that ends with list[i].
For an i of 0, there is only one sequence that ends with list[0] and it's just list[0] itself, which becomes the max:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max
14
There are two sequences ending with list[1] and we get a better max when we use
the one that includes both list[0] and list[1]:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max
14 22
There are three sequences ending in list[2] and the max is achieved by
including all three list values:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max max
14 22 -1
Something very important happens next. There are four different sequences that
end with list[3]. We really have only two choices to consider. We know that
the best you can do with a sequence that ends with list[3] is to get a sum of
-1. That means that if we include that sequence, then we will end up with a
smaller sum than if we were to exclude it. So the best sum can be achieved
with just list[3] itself, without including the values that come before:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max max max
14 22 -1 4
Compare this to what happens later when we are considering what to do for
list[7]:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max max max max max max
14 22 -1 4 10 20 2
Even though the value that comes before list[7] is a negative number, we notice
that the maximum sum that can be achieved that ends with list[6] is a positive
number, so we include it in our answer for the best sum that can be achieved
that ends with list[7]:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max max max max max max max
14 22 -1 4 10 20 2 7
The max is 7, not 5. Continuing this process, we discover the correct sequence
that has a sum of 23:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
max max max max max max max max max max
14 22 -1 4 10 20 2 7 12 23
The key, then, for each value of i is to look to see if it is better to include
the values that came before or whether to start fresh with list[i] only. The
answer is that we include the values that came before if the sum that can be
achieved with them is not negative. Here is the code for the third
approach:
int max = list[0];
int maxStart = 0;
int maxStop = 0;
int start = 0;
int sum = 0;
for (int i = 0; i < list.length; i++) {
if (sum < 0) {
start = i;
sum = 0;
}
sum += list[i];
if (sum > max) {
max = sum;
maxStart = start;
maxStop = i;
}
}
The most frequently executed statements are those in the body of the for loop
(the two if's and the "sum +="). They are each executed n times, so we would
expect this to be an O(n) algorithm.I then switched to a program that has all three of the algorithms I'm going to discuss. It allows you to pick a base value of n and it runs the algorithm for inputs of size n, 2n and 3n, reporting the time each took to execute and the ratio of the various times. We ran it for the first algorithm and asked it to use a base input size of 1500. We got output like this:
How many numbers do you want to use? 1500
Which algorithm do you want to use? 1
for n = 1500, time = 0.913
for n = 3000, time = 7.233
for n = 4500, time = 24.198
Double/single ratio = 7.92223439211391
Triple/single ratio = 26.50383351588171
We're interested in how long it took to execute for each different input size
and the ratios reported at the end.The first algorithm is an O(n3) algorithm, so we were expecting that in doubling the list, the time would increase by a factor of about 8 and in tripling the list, the time would increase by a factor of about 27. The empirical data if fairly close to the prediction.
We found that the second algorithm was much faster than the first. It is an O(n2) algorithm, so we were able to use much larger lists. We ran this second algorithm with a base size of 30,000 and got output like the following.
How many numbers do you want to use? 30000
Which algorithm do you want to use? 2
for n = 30000, time = 0.98
for n = 60000, time = 3.852
for n = 90000, time = 8.642
Double/single ratio = 3.9306122448979592
Triple/single ratio = 8.818367346938775
Again we want to pay attention to the times for the different list sizes and
the ratios at the end. We would predict that time would increase by a factor
of 4 when we double the input and that time would increase by a factor of 9
when we triple the input with this O(n2) algorithm and the empirical
results are fairly close.The third algorithm runs so fast that it almost can't be effectively timed, but here are the results for a run with 5 million elements:
How many numbers do you want to use? 5000000
Which algorithm do you want to use? 3
for n = 5000000, time = 0.017
for n = 10000000, time = 0.033
for n = 15000000, time = 0.046
Double/single ratio = 1.9411764705882353
Triple/single ratio = 2.705882352941176
This is an O(n) algorithm, so we expect that doubling the input will double the
time and tripling the input will triple the time. The empirical results aren't
bad given that the algorithm runs so fast.One of the striking things to notice is that for algorithm 1, we had to limit ourselves to just one thousand elements because it was so slow while with algorithm 3, we could barely time it even with four million values. The moral of the story is that choosing the right algorithm can make a huge difference, particularly if you can choose an algorithm from a better complexity class.
Earlier in the notes I mentioned that counting the most frequently executed line of code "almost" works for predicting the time complexity. There is a notable exception that you have seen. We have often written lines of code like the following:
int[] list = new int[n];
We have seen that when an array is constructed, Java auto-initializes all of
the values to the zero-equivalent of the type. That will require n steps,
which means that this one line of code is O(n). It is one of the few examples
in Java where a single line of code can require time proportional to n.The three algorithms are included in handout 6 and anyone who wants the complete program along with the timing code can download it from the calendar page.