Lecture Notes for Tuesday 2/16/99: Analysis of Algorithms Note on the pace of the course: it's fast, but it won't really get any faster later in the term. Handout: Order of Growth (see web page) Today: Understanding the resource requirements of programs We've been reasoning about programs a little before: [Think computationally] CORRECTNESS - A program that does the wrong thing is utterly useless. - Substitution model - mathematical induction EFFICIENCY A program that does the right thing too slowly isn't so good either. - Analysis of Algorithms We will apply these methods to programs throughout the semester. OVERALL GOALS: Determine the resource requirements of our algorithms. - resources include time, space, money, etc. - usually, we're concerned with time Computational Complexity (named by Hartmanis) Complexity is measured in terms of the SIZE of the input. Traditionally called `n', but you could call it anything. For example, if input is a list of names to sort, then n = number of names in the list. If input is a number, size can either be the *magnitude* of the number or the number of *digits* in it. IT IS IMPORTANT TO KNOW WHICH, they are very different: there are log_10(x) digits for a decimal number of magnitude x. (Try writing 10^20 in unary!) GOAL: To be able to compare the efficiency of algorithms/programs independent of a given machine or a given input. QUEST: Find some function R(n) that measures the resources needed to run it on an input of size n. To simplify the analysis, we investigate *asymptotic* resource requirements -- that is, we "fuzz" the constants to some degree and worry about what happens in the limit of a really, really big input. Functions generally have different asymptotic rates of growth. x and x^2 both go to infinity as x -> infinity, BUT x^2 goes there a lot faster. We can measure this by [note << is scripty-<]: f << g iff f(n) lim ---- = 0 g(n) (limits are to "n to infinity" unless otherwise mentioned.) << = "grows more slowly than" Here are some standard functions (c is any constant): 1 << log n << n^c << c^n << n^n MOTTO: Think BIG. Sometimes you need to take really large n: log n << n^0.0001 Try a = 10^100, we don't have log a < a^0.0001 log a = 100, a^0.0001 is about 1.02 But b = 10^(10^100). Then log b = 10^100, b^0.0001 is about 10^(10^96) [Does anyone know the name for b?] WARNING: You *cannot* generally tell f << g by picking a value for n and trying f(n) < g(n). Only the limit thing will work. We use order of growth notation to capture ``important differences'' in these asymptotic growth rates of functions. ORDER OF GROWTH O(n), O(n^2), etc. "functions of the same order, or rate of growth" f << g is a little too strong: note that x^2 + 1 NOT << x^2 + 2 but these are asymptotically the same [what is the limit?] We could try lim f/g -> C for some constant C; this generalizes <<, and also handles x^2 + 1 vs x^2 + 2 What we do instead is a slightly stronger condition (how so?) DEFINITION: f is O(g) -- pronounced, "f is order g" -- when there is a constant c such that f(n) <= c g(n) for all n >= 1. MEANS: f is at most g. Kind of a <=. EXAMPLES: f(n) = (1/4)n^3 + 27n^2 + n + 1000 f is O(n^3) Could determine that 1/4 n^3 < c n^3 for all n>=1 when c >= 1. 27 n^2 < c n^3 for all n>=1 when c >= 27. etc. THIS is way too much work. We don't need to compute these constants as long as we know they exist. Here are some rules for manipulating O( )-notation: [Proofs: see CS 280, 410, 482] 1. r n^a = O(n^b) whenever a <= b, for any r (note r, a, b independent of n) "Multiplicative constants don't matter" Another consequence: n^2 = O(n^3) as well as O(n^4), in fact its any order larger than O(n^2). We want the LEAST BOUND, which O(n^2) 2. O(f) + O(g) = O(f+g) These two rules let us simplify that polynomial example pretty easily. [Do this in section!] 3. f = O(f) 4. O(a) = O(b) if a and b are any two constants. Just write O(1) for "constant" 5. log_b(n) = O(log n) for ANY b. Note: CSists like their logs to the base 2: log 8 = 3 Mathematicians like base e Engineers like base 10 But, for order of growth, it doesn't matter. They differ by a *constant factor*, and O( )-notation drops constant factors. log_2(n) = log_10(n)*log_2(10) = O(log n) (note log_2(10) is a constant) This stuff is pretty old == 1800's (Euler, I believe) With << and O(.) at our disposal, let's look at the complexity of some functions. Here is the set of definitions I used in class: ----------------------------------------------------------------- (define (length x) (if (empty? x) 0 (+ 1 (length (tail x))))) (define (double x) (append x x)) ;; (length small-list) = 10 (define small-list (list 1 2 3 4 5 6 7 8 9 10)) ;; (length big-list) = 320 (define big-list (double (double (double (double (double small-list)))))) ;; (length very-big-list) = 2560 (define very-big-list (double (double (double big-list)))) ;; Counts down from i to 0 -- used to waste time (define (waste-time i) (if (= i 0) 0 (waste-time (- i 1)))) ;; Sums the numbers in a list, wasting a bit of time before starting. (define (sum-1 x) (let ((y (waste-time 100000))) (define (sum-list z) (if (empty? z) 0 (+ (head z) (sum-list (tail z))))) (sum-list x))) ;; Sums the numbers in a list. (define (sum-2 x) (define (sum-list z) (if (= 0 (length z)) 0 (+ (head z) (sum-list (tail z))))) (sum-list x)) ----------------------------------------------------------------- Which algorithm is better, sum-1 or sum-2? When we run sum-1 on small-list, it takes quite a bit of time, whereas sum-2 runs pretty fast. But when we run sum-1 on very-big-list, it runs a _lot_ faster than sum-2. Claim: the time to run sum-1 on an input list x of length n takes time T_sum-1(n) = k1*n+k2 for some constants k1 and k2. Claim: the time to run sum-1 on an input list x of length n takes time T_sum-2(n) = k3*n^2+k4*n+k5 for some constants k3,k4, and k5. (We'll show you how to calculate this in a second.) So T_sum-1(n) is in O(n) and T_sum-2(n) is in O(n^2). [Why?] Functions in O(n) go to infinity much slower than functions in O(n^2). That is, n 1 lim --- = - <---- not a constant! n^2 n so we know that for some n, T_sum-2 is going to take more time than T_sum-1. And we saw this experimentally. How do we determine what T_sum-1 and T_sum-2 are? -------------------------------------------------------------------- Let's analyze the time it takes to run T_sum-1. In general, what we do is look at how the function executes on an input, assume that built in operations like empty? or + take constant time, and compute an upper bound on how long the function will take. If the function involves recursive calls, then we express the time it takes as a recursive equation (called a recurrence equation) and then attempt to solve the recurrence equation for a closed-form solution. WARNING: most built-in operations like empty? or + take constant time, but not all. For instance, the append operation does not take constant time. When in doubt, ask a TA or consultant or at least state your assumptions. ;; Sums the numbers in a list, wasting a bit of time before starting. (define (sum-1 x) (let ((y (waste-time 100000))) (define (sum-list z) (if (empty? z) 0 (+ (head z) (sum-list (tail z))))) (sum-list x))) On any list x of length n we must: (1) waste-time -- this takes some constant amount of time k1 (2) call sum-list on the argument -- this takes time T_sum-list(n) We can summarize this by saying: T_sum-1(n) = k1 + T_sum-list(n) Now we need to figure out what T_sum-list is as a function of n. If we call sum-list on a list of length 0, then we must: (1) check to see if the list is empty -- this takes a constant time k2 (2) return the value 0 -- this takes a constant time k3 If we call sum-list on a list z of length n+1, then we must: (1) check to see if the list is empty -- this takes a constant time k2 (2) get the head of the list -- constant time k4 (3) get the tail of the list -- constant time k5 (4) call sum-list on the tail of the list: takes time T_sum-list(n) because the tail of z has a length one less than the length of z. (5) add the result to the head -- constant time k6 (6) return the result -- constant time k7 We can summarize this by saying: T_sum-list(0) = k2+k3 T_sum-list(n+1) = k2+k4+k5+T_sum-list(n)+k6+k7 We can rephrase this as: T_sum-list(0) = k8 T_sum-list(n+1) = k9+T_sum-list(n) where k8 = k2+k3, k9=k2+k4+k5+k6+k7. Finally, we can rephrase this as a summation: n ___ T_sum-list(n) = \ k8 + / k9 --- i=1 and collapse this down to: T_sum-list(n) = k8 + n*k9 Therefore the time to run T-sum-1 can be expressed as: T_sum-1(n) = k10 + n*k9 where k10 = k1+k8. Thus, T_sum-1 is in O(n). In summary, sum-1 runs in time that is linear in the size of the input list. -------------------------------------------------------------------- Now let's analyze the time it takes to run T_sum-2. This time, we're going to ignore individual constants because as we saw above, they didn't really matter. (define (length x) (if (empty? x) 0 (+ 1 (length (tail x))))) (define (sum-2 x) (define (sum-list z) (if (= 0 (length z)) 0 (+ (head z) (sum-list (tail z))))) (sum-list x)) The time to run sum-2 on a list x of length n requires that we simply call sum-list on that list. (Note that this is a different sum-list function than the one we analyzed above.) T_sum-2(n) = T_sum-list(n) So now we must analyze the time it takes to run sum-list on a list z of length n. If the list z is empty (i.e., of length 0) then we must: (1) calculate the length of the list: takes time T_length(0) (2) compare the length to 0: takes constant time (3) return 0: takes constant time So: T_sum-list(0) = k3 + T_length(0) where k3 is the constant time for steps 2 and 3. If the list z is of length n+1 then we must: (1) calculate the length of the list: takes time T_length(n+1) (2) compare the length to 0: takes constant time (3) get the head of the list: takes constant time (4) get the tail of the list: takes constant time (5) call sum-list on the tail: takes time T_sum-list(n) (6) add the results: takes constant time So: T_sum-list(n+1) = k8 + T_length(n+1) + T_sum-list(n) where k8 is the constant time for steps 2,3,4, and 6. Now we must analyze the length function to determine how long it takes. It's pretty easy to see that length takes time: T_length(n) = k9*n + k10 for some constants k9 and k10. So: T_sum-list(n+1) = k8 + k9*(n+1) + k10 + T_sum-list(n) We can further simplify this to: T_sum-list(n+1) = k11 + k9*n + T_sum-list(n) Now we can rewrite T_sum-list using a summation: n ---- T_sum-list(n) = k3 + T_length(0) + \ (k9*i + k11) / ---- i = 1 This can be simplified into: n n ---- ---- T_sum-list(n) = k3 + T_length(0) + k9*\ i + \ k11 / / ---- ---- i = 1 i=1 which in turn simplifies to: n ---- T_sum-list(n) = k12 + k11*n + k9*\ i / ---- i = 1 where k12 = k3+T_length(0). Now from class, we know that the sum of the numbers from 1 to n is n*(n+1)/2. So now we can simplify this even more to: T_sum-list(n) = k12 + k13*n + k9*n*(n+1)/2 which after a bit of simplification, becomes: T_sum-list(n) = k12 + k14*n + k15*n^2 Therefore, T_sum-list is in O(n^2). ------------------------------------------------------------------------ Now let's analyze some more interesting functions. Below are two sorting functions, one called insert-sort and the other merge-sort. In class, we saw that insert-sort is much slower than merge-sort for large lists. This is because: T_insert-sort(n) is O(n^2) T_merge-sort(n) is O(n * log(n)) and n*log(n) grows slower than n^2. (Why -- check the limit of the rations.) Let's analyze the two functions to see why they are O(n^2) and O(n*log(n)) respectively. ------------------------------------------------------------------------ ;; Sorts the list of numbers x in ascending order by inserting the ;; elements of the list in proper order into an (initially) empty list. (define (insert-sort x) ;; Inserts the element x into the list y (define (insert x y) (if (empty? y) (list x) (let ((h (head y)) (t (tail y))) (if (<= x h) (cons x y) (cons h (insert x t)))))) ;; Inserts the elements elts into the sorted-list y (define (inserts elts sorted-list) (if (empty? elts) sorted-list (inserts (tail elts) (insert (head elts) sorted-list)))) ;; Sort x by insert each element into an empty list (inserts x empty)) The time to do an insertion sort on a list x of size n is proportional to the time to do inserts on x with an empty list: T_insert-sort(n) = T_inserts(n,0) The time to do inserts can be calculated as: T_inserts(0,m) = k1 T_inserts(n+1,m) = k2 + T_inserts(n,m+1) + T_insert(m) and the time to do insert can be calculated as: T_insert(0) = k3 T_insert(n+1) = k4 + T_insert(n) We've seen the pattern for T_insert before and can immediately solve the recurrence equations to get: T_insert(n) = k4*n+k3 So: T_inserts(0,m) = k1 T_inserts(n+1,m) = k5 + T_inserts(n,m+1) + k4*m Unfortunately, because T_inserts has two arguments, it makes it a little hard to analyze. But we notice that every time we take an element from the first list (elts), we add it to the second list. So, T_inserts has an invariant that, if it's initially called with lists of length n and m, then the recursive call will pass in lists of length n-1 and m+1. Because we initially call inserts with x (a list of length n) and empty (a list of length 0), we know that the size of the two lists must equal n. With this (confusing) information, we can write the time to do an insertion sort as: n ---- T_insert-sort(n)= \ (k5 + k4*(n-i)) / ---- i=1 which simplifies to: n ---- T_insert-sort(n)= k6*n + k4*\ (n-i) / ---- i=1 But the sum from i=1 to n of n-i is equal to the sum from i=1 to n of i! So: n ---- T_insert-sort(n)= k6*n + k4*\ i / ---- i=1 and again, we know that the sum of the numbers from 1 to n is proportional to n^2. So we end up with something like: T_insert-sort(n)= k7*n + k4*n^2 and we can conclude that T_insert-sort is in O(n^2). ------------------------------------------------------------------------ Now let's analyze mergesort: ;; Sorts the list of numbers x in ascending order by splitting the ;; list in half, sorting the two sub-lists, and then merging the two ;; sorted lists together. (define (merge-sort x) ;; Merges two sorted lists. (define (merge list1 list2) (if (empty? list1) list2 (if (empty? list2) list1 (let ((h1 (head list1)) (h2 (head list2))) (if (<= h1 h2) (cons h1 (merge (tail list1) list2)) (cons h2 (merge list1 (tail list2)))))))) ;; Splits the list x into two lists list1 and list2 (define (split-list x list1 list2) (if (empty? x) (merge (merge-sort list1) (merge-sort list2)) (split-list (tail x) list2 (cons (head x) list1)))) ;; If the list is empty or contains only one element, then we're done. ;; Otherwise, split into two lists, sort them, and then merge back. (if (or (empty? x) (empty? (tail x))) x (split-list x empty empty))) T_merge-sort(0) = k1 T_merge-sort(1) = k2 T_merge-sort(n+1) = k2 + T_split-list(n+1,0,0) So the time to do a merge sort is at least the time to do a T_split-list. At the end of the split-list, list1 has half the elements and list2 has half the elements. It takes time proportional to the original list to split them. So T_split-list(n,0,0) = k3*n + k4*T_merge(n/2,n/2) + k5*T_merge-sort(n/2) + k5*T_merge-sort(n/2) The time it takes to do a merge is at worst the sum of the lengths of the lists: T_merge(n,m) = k6*(n+m) So we have: T_split-list(n,0,0) = k7*n + k5*T_merge-sort(n/2) + k5*T_merge-sort(n/2) = k7*n + k8*T_merge-sort(n/2) Therefore, we have the equation: T_merge-sort(n) = k9*n + k8*T_merge-sort(n/2) when n > 1. To summarize, we have as our recurrence equations for merge-sort: T_merge-sort(0) = k1 T_merge-sort(1) = k2 T_merge-sort(n) = k9*n + k8*T_merge-sort(n/2) (n > 1) How do we solve this system of equations? Well, at each step we're doing work proportional to the size of the list (n) and each time, we cut the list in half -- so what we need to do is figure out how many times can we cut the list in half until we're down to a base case. The answer is that, if we have a list of length n, then we can only cut it in half log_2(n) times before we reach a list of size 1 or 0. In general, when we have recurrence equations of the form: F(0) = k1 F(1) = k2 F(n) = k3 + F(n/2) (for n > 1) they have a solution bounded by: F(n) <= k2 + c * log_2(n) for some c. For instance, when n=8, when we have: F(8) = k3 + F(4) = k3 + k3 + F(2) = k3 + k3 + k3 + F(1) = k3 + k3 + k3 + k2 = k2 + k3*3 = k2 + k3*log_2(8) So, because we have: T_merge-sort(0) = T_merge-sort(1) = k2 T_merge-sort(n) = k9*n + k8*T_merge-sort(n/2) (n > 1) we can conclude that: T_merge-sort(n) = c1 * (n * log_2(n)) + c2 for some constants c1 and c2. Therefore, T_merge-sort is in O(n*log(n)).