The Perceptron

Cornell CS 4/5780

Fall 2023

Assumptions

Binary classification (i.e. $y_{i} \in {- 1, + 1}$ )
Data is linearly separable

Classifier

h (x_{i}) = sign (w^{⊤} x_{i} + b)

b

is the bias term (without the bias term, the hyperplane that

w

defines would always have to go through the origin). Dealing with

b

can be a pain, so we 'absorb' it into the feature vector

w

by adding one additional constant dimension. Under this convention,

x_{i} becomes [\begin{matrix} x_{i} \\ 1 \end{matrix}] w becomes [\begin{matrix} w \\ b \end{matrix}]

We can verify that

{[\begin{matrix} x_{i} \\ 1 \end{matrix}]}^{⊤} [\begin{matrix} w \\ b \end{matrix}] = w^{⊤} x_{i} + b

Using this, we can simplify the above formulation of

h (x_{i})

h (x_{i}) = sign (w^{⊤} x)

(Left:) The original data is 1-dimensional (top row) or 2-dimensional (bottom row). There is no hyper-plane that passes through the origin and separates the red and blue points. (Right:) After a constant dimension was added to all data points such a hyperplane exists.

Observation: Note that

y_{i} (w^{⊤} x_{i}) > 0 ⟺ x_{i} is classified correctly

where 'classified correctly' means that

x_{i}

is on the correct side of the hyperplane defined by

w

. Also, note that the left side depends on

y_{i} \in {- 1, + 1}

(it wouldn't work if, for example

y_{i} \in {0, + 1}

Perceptron Algorithm

Now that we know what the

w

is supposed to do (defining a hyperplane the separates the data), let's look at how we can get such

w

. Perceptron Algorithm

Geometric Intuition

Illustration of a Perceptron update. (Left:) The hyperplane defined by $w_{t}$ misclassifies one red (-1) and one blue (+1) point. (Middle:) The red point $x$ is chosen and used for an update. Because its label is -1 we need to subtract $x$ from $w_{t}$ . (Right:) The udpated hyperplane $w_{t + 1} = w_{t} - x$ separates the two classes and the Perceptron algorithm has converged.

Quiz: Assume a data set consists only of a single data point

{(x, + 1)}

. How often can a Perceptron misclassify this point

x

repeatedly? What if the initial weight vector

w

was initialized randomly and not as the all-zero vector?

Perceptron Convergence

The Perceptron was arguably the first algorithm with a strong formal guarantee. If a data set is linearly separable, the Perceptron will find a separating hyperplane in a finite number of updates. (If the data is not linearly separable, it will loop forever.)

The argument goes as follows: Suppose $\exists w^{*}$ such that $y_{i} (x^{⊤} w^{*}) > 0$ $\forall (x_{i}, y_{i}) \in D$ . Now, suppose that we rescale each data point and the $w^{*}$ such that $| | w^{*} | | = 1 and | | x_{i} | | \leq 1 \forall x_{i} \in D$ Let us define the Margin $γ$ of the hyperplane $w^{*}$ as $γ = min_{(x_{i}, y_{i}) \in D} | x_{i}^{⊤} w^{*} |$ .

A little observation (which will come in very handy): For all $x$ we must have $y (x^{⊤} w^{*}) = | x^{⊤} w^{*} | \geq γ$ . Why? Because $w^{*}$ is a perfect classifier, so all training data points $(x, y)$ lie on the "correct" side of the hyper-plane and therefore $y = s i g n (x^{⊤} w^{*})$ . The second inequality follows directly from the definition of the margin $γ$ .

To summarize our setup:

All inputs $x_{i}$ live within the unit sphere
There exists a separating hyperplane defined by $w^{*}$ , with $∥ w ∥^{*} = 1$ (i.e. $w^{*}$ lies exactly on the unit sphere).
$γ$ is the distance from this hyperplane (blue) to the closest data point.

Theorem: If all of the above holds, then the Perceptron algorithm makes at most

1 / γ^{2}

mistakes. Proof:
Keeping what we defined above, consider the effect of an update (

w

becomes

w + y x

) on the two terms

w^{⊤} w^{*}

and

w^{⊤} w

. We will use two facts:

$y (x^{⊤} w) \leq 0$ : This holds because $x$ is misclassified by $w$ - otherwise we wouldn't make the update.
$y (x^{⊤} w^{*}) > 0$ : This holds because $w^{*}$ is a separating hyper-plane and classifies all points correctly.

Consider the effect of an update on $w^{⊤} w^{*}$ : $(w + y x)^{⊤} w^{*} = w^{⊤} w^{*} + y (x^{⊤} w^{*}) \geq w^{⊤} w^{*} + γ$ The inequality follows from the fact that, for $w^{*}$ , the distance from the hyperplane defined by $w^{*}$ to $x$ must be at least $γ$ (i.e. $y (x^{⊤} w^{*}) = | x^{⊤} w^{*} | \geq γ$ ). This means that for each update, $w^{⊤} w^{*}$ grows by at least $γ$ .
Consider the effect of an update on $w^{⊤} w$ : $(w + y x)^{⊤} (w + y x) = w^{⊤} w + \underset{< 0}{\underset{⏟}{2 y (w^{⊤} x)}} + \underset{0 \leq \leq 1}{\underset{⏟}{y^{2} (x^{⊤} x)}} \leq w^{⊤} w + 1$ The inequality follows from the fact that
- $2 y (w^{⊤} x) < 0$ as we had to make an update, meaning $x$ was misclassified
- $0 \leq y^{2} (x^{⊤} x) \leq 1$ as $y^{2} = 1$ and all $x^{⊤} x \leq 1$ (because $∥ x ∥ \leq 1$ ).

This means that for each update,

w^{⊤} w

grows by at most 1

Now remember from the Perceptron algorithm that we initialize $w = 0$ . Hence, initially $w^{⊤} w = 0$ and $w^{⊤} w^{*} = 0$ and after $M$ updates the following two inequalities must hold:
- (1) $w^{⊤} w^{*} \geq M γ$
- (2) $w^{⊤} w \leq M$ .
We can then complete the proof: $\begin{aligned} M γ & \leq w^{⊤} w^{*} & By (1) \\ = ∥ w ∥ \cos (θ) & by definition of inner-product, where θ is the angle between w and w^{*} . \\ \leq | | w | | & by definition of \cos, we must have \cos (θ) \leq 1 . \\ = \sqrt{w^{⊤} w} & by definition of ∥ w ∥ \\ \leq \sqrt{M} & By (2) \\ \Rightarrow M γ \leq \sqrt{M} \\ \Rightarrow M^{2} γ^{2} \leq M \\ \Rightarrow M \leq \frac{1}{γ^{2}} & And hence, the number of updates M is bounded from above by a constant. \end{aligned}$

Quiz: Given the theorem above, what can you say about the margin of a classifier (what is more desirable, a large margin or a small margin?) Can you characterize data sets for which the Perceptron algorithm will converge quickly? Draw an example.

History

Initially, huge wave of excitement ("Digital brains") (See The New Yorker December 1958)
Then, contributed to the A.I. Winter. Famous example of a simple non-linearly separable data set, the XOR problem (Minsky 1969):