I constrasted this with the way that we wrote the indexOf method for both the ArrayIntList and the LinkedIntList. In both cases we iterated through the list from beginning to end until we either found the value or ran out of values. This is an approach known as linear search.
I said that clearly the binary search is faster than the linear search, but how much faster? Computer scientists answer questions like these by talking about the complexity of an algorithm.
The word "complexity" can be interpreted in many ways. It sounds like a measure of how complex or how complicated a program is. jGRASP has a tool under the File menu that allows you to create a "Complexity Profile Graph" of your code, which is a software engineering concept that is somewhat similar to this. But that's not how computer scientists use the term most often. When we refer to the complexity of an algorithm or a code fragment, we most often are referring to the resources that it requires to execute. The two resources that we are generally most interested in are:
Of these two, the resource that computer scientists most often refer to when talking about complexity is time. In particular, we are interested in the growth rate as the input size increases. We begin by deciding on some way to measure the size of the input (e.g., the number of names to sort, the number of numbers to examine, etc) and call this "n". We are interested in what happens when we change n. For example, if it takes time "t" to execute for n items, how much time does it take to execute for 2n items?
I pointed out that this is one of the few places where computer science is actually like a science. Some instructors ask their students to collect empirical timing data for different input sizes and have them plot these values to see if the plot matches the prediction. Unfortunately, these experiments are more difficult to perform on modern computers because features like cache memory skew the results. The important thing is that the predictions hold for large values of n.
I then mentioned a simple rule of thumb that you can apply to Java programs to figure out the complexity of a code segment. I mentioned that it "almost" works. The idea is to find the line of code that is executed most often. In thinking about this, you have to be careful how you count. For example, with a for loop, we'd count the loop itself as executing just once, but the statements controlled by the loop might be executed many times. Of course, a for loop can be inside a for loop in which case the inner loop is executed multiple times. But think in terms of how many times you enter the loop when counting the number of executions of the line of code that begins with "for".
I pointed out that I see a lot of undergraduates who obsess about efficiency and I think that in general it's a waste of their time. Many computer scientists have commented that premature optimization is counterproductive. Don Knuth has said that "Premature optimization is the root of all evil." The key is to focus your attention where you can get real benefit. The analogy I'd make is that if you found that a jet airplane weighed too much, you might decide to put the pilot on a diet. While it's true that in some sense every little bit helps, you're not going to make much progress by trying to slim down the pilot when the plane and its passengers and cargo weigh so much more than the pilot. I see lots of undergraduates putting the pilot on a diet while ignoring much more important details.
In terms of the growth rate of different algorithms, I mentioned that some of your intuitions from calculus will be helpful. You've probably been asked to solve problems like figuring out what the limit is as you approach infinity of an expression like this:
n^3 - 18 n^2 + 385 n + 708
--------------------------
0.005 n^4 - 13 n^2 + 73842
When you solve a limit like this, you ignore things like coefficients and you
ignore small terms. What matters here is that you basically have:
n^3
---
n^4
The rest is noise. So this is something that you know is going to approach 0
because eventually the n^4 will dominate the n^3 no matter what the
coefficients and lower-order terms are. We use similar reasoning with
complexity. We ignore constant multipliers and we ignore lower order terms to
focus on the main term.I said that different algorithms naturally fall into different complexity classes:
The following table presents several hypothetical algorithm runtimes as an input size N grows, assuming that each algorithm required 100ms to process 100 elements. Notice that even if they all start at the same runtime for a small input size, the ones in higher complexity classes become so slow as to be impractical.Then I talked about a specific problem. The idea is that we have a list of integers, both positive and negative, and we want to find the subsequence that has the highest sum. If there weren't any negative integers, you'd always include all of the numbers. But because some of them can be negative, it might be the case that some portion of the list has a sum that is greater than any other sequence from the list. The subsequences always involve taking a contiguous chunk of the list. This particular problem has often been used by Microsoft as an interview question, probably because there are different ways to solve it, some of which are much faster than others.
input size (N) O(1) O(log N) O(N) O(N log N) O(N2) O(N3) O(2N) 100 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 200 100 ms 115 ms 200 ms 240 ms 400 ms 800 ms 32.7 sec 400 100 ms 130 ms 400 ms 550 ms 1.6 sec 6.4 sec 12.4 days 800 100 ms 145 ms 800 ms 1.2 sec 6.4 sec 51.2 sec 36.5 million years 1600 100 ms 160 ms 1.6 sec 2.7 sec 25.6 sec 6 min 49.6 sec 42.1 * 1024 years 3200 100 ms 175 ms 3.2 sec 6 sec 1 min 42.4 sec 54 min 36 sec 5.6 * 1061 years
As an example, suppose you have an array that stores these values:
[0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| 14 | 8 | -23 | 4 | 6 | 10 | -18 | 5 | 5 | 11 |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
The maximum sum is obtained by adding up the values from index [3] through
[9]:
4 + 6 + 10 + -18 + 5 + 5 + 11 = 23
It might seem odd that you include -18, but it's because including that allows
you to include the three numbers that come before, which add up to 20 (a net
gain of 2 for the overall sum).There is a simple way to solve this that involves picking each possible subsequence. We can have one for loop that generates each possible starting point and another for loop that generates each possible stopping point:
for (int start = 0; start < list.length; start++) {
for (int stop = start; stop < list.length; stop++) {
look at the numbers from start to stop
}
}
So how do we "look at the numbers from start to stop"? We can write a loop
that adds up each of those numbers:
int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i];
}
And once we have that sum, we can compare it against the maximum sum we've seen
so far and reset the maximum if this sum is better:
if (sum > max) {
max = sum;
}
Putting these pieces together and including some initialization outside the
loop, we end up with the following code:
int max = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++)
for (int stop = start; stop < list.length; stop++) {
int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i];
}
if (sum > max) {
max = sum;
maxStart = start;
maxStop = stop;
}
}
That's the first approach. The line that is executed most often in this
approach is the "sum += ..." line inside the innermost for loop (the "i" loop
that adds up the list). It is nested inside three different loops, each of
which executes on the order of n times. So we would predict that this code
would be an O(n3) algorithm.Then I asked how the algorithm could be improved. How can we do this faster? The bottleneck is the line that is adding up individual numbers and the key to improving the algorithm is noticing how we're doing a lot of duplicate work. Think about what happens the first time through the outer loop when "start" is equal to 0. We go through the inner loop for all possible values of "stop". So suppose the list is 2000 long. We're going to compute:
the sum from 0 to 0
the sum from 0 to 1
the sum from 0 to 2
the sum from 0 to 3
the sum from 0 to 4
...
the sum from 0 to 1999
Those are all the possibilities that start with 0. We have to explore each of
these possibilities, but think about how we're computing the sums. We have an
inner "i" loop that is computing the sum from scratch each time. For example,
suppose that we just finished computing the sum from 0 to 1000. We next
compute the sum from 0 to 1001. But we start from the very beginning and have
i go through all of the values 0 through 1001 when we've just computed the sum
from 0 to 1000. That was a lot of work. Then you throw away that sum and
start from scratch to add up the values from 0 to 1001. But why start back at
the beginning? If you know what the values from 0 to 1000 add up to, then just
add the value at position 1001 to get the sum from 0 to 1001.So the key is to eliminate the inner "i" loop by keeping a running sum. This requires us to move the initialization of sum from the inner loop to the outer loop so that we don't forget the work we've already done.
int max = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++) {
int sum = 0;
for (int stop = start; stop < list.length; stop++) {
sum += list[stop];
if (sum > max) {
max = sum;
maxStart = start;
maxStop = stop;
}
}
}
In this code the most frequently executed statements are inside the for loop
for "stop" (the line that begins "sum +=" and the if). There are other lines
of code that you could argue also tie with this, like the for loop test and for
loop increment. Each of these lines of code is inside an outer loop that
iterates n times and an inner loop that iterates an average of n/2 times. So
this is an O(n2) algorithm.I mentioned that there is a third algorithm, although I wouldn't have time to discuss it in detail and it is the most difficult to understand, so I did not attempt to prove its correctness. I did, however, try to explain the basic idea. The key is to avoid computing all of the sums. We want to have some heuristic that would allow us to ignore certain possibilities. We do that with a single loop and by keeping track of the highest possible sum you can form that ends with list[i].
So suppose that we are considering the i-th value in the list for some i greater than 0. Let's say i is 10. Think about subsequences that end with list[10]. Some of them begin with list[10] and others begin earlier, including list[9] and potentially other values that appear before list[10]. Under what circumstances would we get a higher sum by starting with list[10] versus including these earlier values? That's the key question.
The answer is that if the best subsequence you can find ending in list[9] adds up to a positive number, then that sequence is worth including. If they add up to a negative number, then they are taking away from the sum we are trying to generate. In other words, if those earlier values add up to a negative number, then we can get a higher sum by excluding them and starting our sequence with list[10].
Here is the code for the third approach:
int max = list[0];
int maxStart = 0;
int maxStop = 0;
int start = 0;
int sum = 0;
for (int i = 0; i < list.length; i++) {
if (sum < 0) {
start = i;
sum = 0;
}
sum += list[i];
if (sum > max) {
max = sum;
maxStart = start;
maxStop = i;
}
}
The most frequently executed statements are those in the body of the for loop
(the two if's and the "sum +="). They are each executed n times, so we would
expect this to be an O(n) algorithm.I then switched to a program that has all three of the algorithms I'm going to discuss. It allows you to pick a base value of n and it runs the algorithm for inputs of size n, 2n and 3n, reporting the time each took to execute and the ratio of the various times. We ran it for the first algorithm and asked it to use a base input size of 1500. We got output like this:
How many numbers do you want to use? 1500
Which algorithm do you want to use? 1
for n = 1500, time = 0.913
for n = 3000, time = 7.233
for n = 4500, time = 24.198
Double/single ratio = 7.92223439211391
Triple/single ratio = 26.50383351588171
We're interested in how long it took to execute for each different input size
and the ratios reported at the end.The first algorithm is an O(n3) algorithm, so we were expecting that in doubling the list, the time would increase by a factor of about 8 and in tripling the list, the time would increase by a factor of about 27. The empirical data if fairly close to the prediction.
We found that the second algorithm was much faster than the first. It is an O(n2) algorithm, so we were able to use much larger lists. We ran this second algorithm with a base size of 30,000 and got output like the following.
How many numbers do you want to use? 30000
Which algorithm do you want to use? 2
for n = 30000, time = 0.98
for n = 60000, time = 3.852
for n = 90000, time = 8.642
Double/single ratio = 3.9306122448979592
Triple/single ratio = 8.818367346938775
Again we want to pay attention to the times for the different list sizes and
the ratios at the end. We would predict that time would increase by a factor
of 4 when we double the input and that time would increase by a factor of 9
when we triple the input with this O(n2) algorithm and the empirical
results are fairly close.The third algorithm runs so fast that it almost can't be effectively timed, but here are the results for a run with 5 million elements:
How many numbers do you want to use? 5000000
Which algorithm do you want to use? 3
for n = 5000000, time = 0.017
for n = 10000000, time = 0.033
for n = 15000000, time = 0.046
Double/single ratio = 1.9411764705882353
Triple/single ratio = 2.705882352941176
This is an O(n) algorithm, so we expect that doubling the input will double the
time and tripling the input will triple the time. The empirical results aren't
bad given that the algorithm runs so fast.One of the striking things to notice is that for algorithm 1, we had to limit ourselves to just one thousand elements because it was so slow while with algorithm 3, we could barely time it even with four million values. The moral of the story is that choosing the right algorithm can make a huge difference, particularly if you can choose an algorithm from a better complexity class.
The three algorithms are included in handout 9 and anyone who wants the complete program along with the timing code can download it from the calendar page.