Note on the pace of the course: it's fast, but it won't really get any faster later in the term.
PS#1 stats, Perfect snowflake winning entries are on the web. People did very well.
Perfect snowflake winners: Dion Harmon and Yisha Zhang.
Handout
Today
Mathematical basis, experimentally justified.
We've been reasoning about programs a little before:
[Think computationally]
- A program that does the wrong thing is utterly useless.
- Substitution model
- mathematical induction
- Analysis of Algorithms
We will apply these methods to programs throughout the semester.
OVERALL GOALS:
Complexity is measured in terms of the SIZE of the input.
n = number of names in the list.
For today we'll do the analysis in terms of the magnitude of the number itself
GOAL: To be able to compare the efficiency of algorithms/programs independent of a given machine or a given input.
QUEST:
Find some function R(n) that measures the resources needed to run it on an input of size n.
To simplify the analysis, we investigate asymptotic resource requirements.
Example of an asymptote:
y = sqrt(1+x^2)
x = 0, y = 1
x = 1, y = sqrt(2) ~= 1.4
x = 10, y = 10.05
x = 100, y = 100.005
x = 1000, y = 1000.0005, you get the point
>>> x=-y.
The lines y=x and y=-x are asymptotes:
sqrt(1+x^2) ---> x as x ---> infinity.
We generalize this notion of asymptote to mean
Functions generally have different asymptotic rates of growth.
x and x^2 both go to infinity as x --> infinity,
BUT
x^2 goes there a lot faster.
We can measure this by [note << is scripty-<]:
f << g iff
lim (f(n) / g(n)) = 0
(limits are to "n to infinity" unless otherwise mentioned.)
<< = "grows more slowly than"
Here are some standard functions (c is any constant):
1 << log n << n^c << c^n << n^n
REMINDER:
Think BIG. Sometimes you need to take really large n:
log n << n^0.0001
Try a = 10^100, we don't have log a < a^0.0001
log a = 100, a^0.0001 is about 1.02
But b = 10^(10^100). Then
log b = 10^100, b^0.0001 is about 10^(10^96)
[Does anyone know the name for b? Or 10^b?]
WARNING:
You cannot generally tell f<<g by picking a value for n and trying f(n) < g(n)
Only the limit thing will work..
We use order of growth notation to capture ``important differences'' in these asymptotic growth rates of functions.
ORDER OF GROWTH
O(n), O(n^2), etc. "functions of the same order, or rate of growth"
f << g is a little too strong: note that x^2 + 1 NOT << x^2 + 2
but these are asymptotically the same [what is the limit?]
We could try lim f/g -> C for some constant C; this generalizes <<, and also handles
x^2 + 1 vs x^2 + 2
What we do instead is a slightly stronger condition (how so?)
DEF:
f is O(g) -- pronounced, "f is order g" -- when there is a constant c such that
f(n) <= c g(n) for all n >= 1.
MEANS:
f is at most g. Kind of a <=.
WARNING:
Often written f = O(g)
This is an abuse of `='. This is in NO WAY an equality, and
careful people write it
f is in O(g)
EXAMPLES:
f(n) = (1/4)n^3 + 27n^2 + n + 1000
f is O(n^3)
Could determine that
1/4 n^3 < c n^3 for all n>=1 when c >= 1.
27 n^2 < c n^3 for all n>=1 when c >= 27.
etc.
THIS is too much work. We don't need to compute these constants as long as we know they exist.
Here are some rules for manipulating O()-notation:
[Proofs: see CS 280 and CS 410]
1. r n^a = O(n^b) whenever a <= b, for any r (note r, a, b independent of n)
"Multiplicative constants don’t matter"
Another consequence: n^2 = O(n^3) as well as O(n^4), in fact its any order larger than O(n^2). We want the LEAST BOUND, which O(n^2)
2. O(f) + O(g) = O(f+g)
These two rules let us simplify that polynomial example pretty easily. [Do this in section!]
3. f = O(f)
4. O(a) = O(b) if a and b are any two constants.
Just write O(1) for "essentially constant"
5. log_b(n) = O(log n) for ANY b (independent of n).
Note: CSists usually take logs to the base 2:
log 8 = 3
Mathematicians like base e
Engineers I think like base 10
But, for order of growth, it doesn't matter.
They differ by a constant factor, and O()-notation drops constant factors.
log_2(n) = log_10(n) * log_2(10) is O(log n)
This stuff is pretty old == 1800's (Euler, I believe)
With << and O(.) at our disposal, let's look at the complexity of
times-1 from a recent lecture:
(define (times-1 a b) (if (= b 0) 0 (+ a (times-1 a (- b 1)))))
>> Recall that this is a recursive process
Let's compute the order of growth as a function of the magnitude of b (not the number of digits).
The operations involved are:
Most built-in primitives are CONSTANT cost, O(1).
[A point that some students forget on problem sets and exams.
We'll show you some a few lectures from today.]
The big question is, how many times does times-1 get called?
(You could back this up by working with the substitution model if you like.)
T(b) = T(b-1) + c ---- if b>0
T(0) = c
c's are just O(1)'s, constants or about that.
These equations are RECURRENCE RELATIONS
We can sometimes solve these:
T(b) = T(b-1) + c
= T(b-2) + c + c
= ...
= T(b-b) + c + c ... + c ;; b c's
= T(0) + bc
= (b+1)c
But we usually don't care about the solutions at this level of detail, since we're just going to blur all the details with O()-notation anyways:
(b+1)c = O(b)
Why? Because we are interested in the complexity as a function of b! It's cb+c and we basically ignore constant summands and factors.
For this class, it'll help to learn some recurrence patterns
Pattern 1
T(n)=T(n-k)+c
T(0)=c
is O(n), where k, c are constants
Here's another one:
(define fast-times (lambda (a b) (cond ((= b 0) 0) ((even? b) (fast-times (double a) (halve b))) (else (+ a (fast-times a (- b 1)))))))
even?, =, double, halve, +, 1 are all constant-time (why?)
T(0) = c
T(b) = T(b/2) + c if b is even > 0
T(b) = T(b-1) + c if b is odd
Consider the easy case, b is a power of 2.
b = 2 ^ (log_2 b)
T(b) = T(b/2) + c
= T(b/4) + c + c
= T(b/8) + c + c + c
..
= T(1) + (log_2 b) * c
= T(0) + (1 + log_2 b) * c
= (2+log_2 b) * c
= O(log b)
This general pattern
>> DIVIDE AND CONQUER << [see CS410]
It typically has logarithmic growth, which is a very good thing:
O(log n) is much smaller than O(n)
log_10(10^100) = 100
10^100 = 10,000,000,000...
Pattern 2
T(n)=T(n/k)+c
T(0)=c
is O(log n)
We still have to finish the analysis for fast-times when n is not a power of two.
So, we're estimating all over the place.
NO: it's a good model.
Running times for fast-times and times-1 on an antique Sparc:
times-1 | fast-times | b |
0.017 | 0 | 2 |
0.017 | 0.033 | 10 |
0.25 | 0.05 | 100 |
2.45 | 0.067 | 1000 |
24.283 | 0.067 | 10000 |
Analyzing running time of our programs (asymptotic analysis, as n gets large):