$$
h(x_i) = \textrm{sign}(\mathbf{w}^\top \mathbf{x}_i + b)
$$
$b$ is the bias term (without the bias term, the hyperplane that $\mathbf{w}$ defines would always have to go through the origin).
Dealing with $b$ can be a pain, so we 'absorb' it into the feature vector $\mathbf{w}$ by adding one additional constant dimension.
Under this convention,
$$
\mathbf{x}_i \hspace{0.1in} \text{becomes} \hspace{0.1in} \begin{bmatrix} \mathbf{x}_i \\ 1 \end{bmatrix} \\
\mathbf{w} \hspace{0.1in} \text{becomes} \hspace{0.1in} \begin{bmatrix} \mathbf{w} \\ b \end{bmatrix} \\
$$
We can verify that
$$
\begin{bmatrix} \mathbf{x}_i \\ 1 \end{bmatrix}^\top \begin{bmatrix} \mathbf{w} \\ b \end{bmatrix} = \mathbf{w}^\top \mathbf{x}_i + b
$$
Using this, we can simplify the above formulation of $h(\mathbf{x}_i)$ to
$$
h(\mathbf{x}_i) = \textrm{sign}(\mathbf{w}^\top \mathbf{x})
$$
(Left:) The original data is 1-dimensional (top row) or 2-dimensional (bottom row). There is no hyper-plane that passes through the origin and separates the red and blue points. (Right:) After a constant dimension was added to all data points such a hyperplane exists.
Observation: Note that
$$
y_i(\mathbf{w}^\top \mathbf{x}_i) > 0 \Longleftrightarrow \mathbf{x}_i \hspace{0.1in} \text{is classified correctly}
$$
where 'classified correctly' means that $x_i$ is on the correct side of the hyperplane defined by $\mathbf{w}$.
Also, note that the left side depends on $y_i \in \{-1, +1\}$ (it wouldn't work if, for example $y_i \in \{0, +1\}$).
Perceptron Algorithm
Now that we know what the $\mathbf{w}$ is supposed to do (defining a hyperplane the separates the data), let's look at how we can get such $\mathbf{w}$.
Perceptron Algorithm
Geometric Intuition
Illustration of a Perceptron update. (Left:) The hyperplane defined by $\mathbf{w}_t$ misclassifies one red (-1) and one blue (+1) point. (Middle:) The red point $\mathbf{x}$ is chosen and used for an update. Because its label is -1 we need to subtract $\mathbf{x}$ from $\mathbf{w}_t$. (Right:) The udpated hyperplane $\mathbf{w}_{t+1}=\mathbf{w}_t-\mathbf{x}$ separates the two classes and the Perceptron algorithm has converged.
Quiz: Assume a data set consists only of a single data point $\{(\mathbf{x},+1)\}$. How often can a Perceptron misclassify this point $\mathbf{x}$ repeatedly? What if the initial weight vector $\mathbf{w}$ was initialized randomly and not as the all-zero vector?
Perceptron Convergence
The Perceptron was arguably the first algorithm with a strong formal guarantee. If a data set is linearly separable, the Perceptron will find a separating hyperplane in a finite number of updates. (If the data is not linearly separable, it will loop forever.)
The argument goes as follows:
Suppose $\exists \mathbf{w}^*$ such that $y_i(\mathbf{x}^\top \mathbf{w}^* ) > 0 $ $\forall (\mathbf{x}_i, y_i) \in D$.
Now, suppose that we rescale each data point and the $\mathbf{w}^*$ such that
$$
||\mathbf{w}^*|| = 1 \hspace{0.3in} \text{and} \hspace{0.3in} ||\mathbf{x}_i|| \le 1 \hspace{0.1in} \forall \mathbf{x}_i \in D
$$
Let us define the Margin $\gamma$ of the hyperplane $\mathbf{w}^*$ as
$
\gamma = \min_{(\mathbf{x}_i, y_i) \in D}|\mathbf{x}_i^\top \mathbf{w}^* |
$.
A little observation (which will come in very handy): For all $\mathbf{x}$ we must have $y(\mathbf{x}^\top \mathbf{w}^*)=|\mathbf{x}^\top \mathbf{w}^*|\geq \gamma$. Why? Because $\mathbf{w}^*$ is a perfect classifier, so all training data points $(\mathbf{x},y)$ lie on the "correct" side of the hyper-plane and therefore $y=sign(\mathbf{x}^\top \mathbf{w}^*)$. The second inequality follows directly from the definition of the margin $\gamma$.
To summarize our setup:
All inputs $\mathbf{x}_i$ live within the unit sphere
There exists a separating hyperplane defined by $\mathbf{w}^*$, with $\|\mathbf{w}\|^*=1$ (i.e. $\mathbf{w}^*$ lies exactly on the unit sphere).
$\gamma$ is the distance from this hyperplane (blue) to the closest data point.
Theorem: If all of the above holds, then the Perceptron algorithm makes at most $1 / \gamma^2$ mistakes.
Proof:
Keeping what we defined above, consider the effect of an update ($\mathbf{w}$ becomes $\mathbf{w}+y\mathbf{x}$) on the two terms $\mathbf{w}^\top \mathbf{w}^*$ and $\mathbf{w}^\top \mathbf{w}$.
We will use two facts:
$y( \mathbf{x}^\top \mathbf{w})\leq 0$: This holds because $\mathbf x$ is misclassified by $\mathbf{w}$ - otherwise we wouldn't make the update.
$y( \mathbf{x}^\top \mathbf{w}^*)>0$: This holds because $\mathbf{w}^*$ is a separating hyper-plane and classifies all points correctly.
Consider the effect of an update on $\mathbf{w}^\top \mathbf{w}^*$:
$$
(\mathbf{w} + y\mathbf{x})^\top \mathbf{w}^* = \mathbf{w}^\top \mathbf{w}^* + y(\mathbf{x}^\top \mathbf{w}^*) \ge \mathbf{w}^\top \mathbf{w}^* + \gamma
$$
The inequality follows from the fact that, for $\mathbf{w}^*$, the distance from the hyperplane defined by $\mathbf{w}^*$ to $\mathbf{x}$ must be at least $\gamma$ (i.e. $y (\mathbf{x}^\top \mathbf{w}^*)=|\mathbf{x}^\top \mathbf{w}^*|\geq \gamma$).
This means that for each update, $\mathbf{w}^\top \mathbf{w}^*$ grows by at least $\gamma$.
Consider the effect of an update on $\mathbf{w}^\top \mathbf{w}$:
$$
(\mathbf{w} + y\mathbf{x})^\top (\mathbf{w} + y\mathbf{x}) = \mathbf{w}^\top \mathbf{w} + \underbrace{2y(\mathbf{w}^\top\mathbf{x})}_{<0} + \underbrace{y^2(\mathbf{x}^\top \mathbf{x})}_{0\leq \ \ \leq 1} \le \mathbf{w}^\top \mathbf{w} + 1
$$
The inequality follows from the fact that
$2y(\mathbf{w}^\top \mathbf{x}) < 0$ as we had to make an update, meaning $\mathbf{x}$ was misclassified
$0\leq y^2(\mathbf{x}^\top \mathbf{x}) \le 1$ as $y^2 = 1$ and all $\mathbf{x}^\top \mathbf{x}\leq 1$ (because $\|\mathbf x\|\leq 1$).
This means that for each update, $\mathbf{w}^\top \mathbf{w}$ grows by at most 1.
Now remember from the Perceptron algorithm that we initialize $\mathbf{w}=\mathbf{0}$. Hence, initially $\mathbf{w}^\top\mathbf{w}=0$ and $\mathbf{w}^\top\mathbf{w}^*=0$ and after $M$ updates the following two inequalities must hold:
(1) $\mathbf{w}^\top\mathbf{w}^*\geq M\gamma$
(2) $\mathbf{w}^\top \mathbf{w}\leq M$.
We can then complete the proof:
\begin{align}
M\gamma &\le \mathbf{w}^\top \mathbf{w}^* &&\text{By (1)} \\
&=\|\mathbf{w}\|\cos(\theta) && \text{by definition of inner-product, where $\theta$ is the angle between $\mathbf{w}$ and $\mathbf{w}^*$.}\\
&\leq ||\mathbf{w}|| &&\text{by definition of $\cos$, we must have $\cos(\theta)\leq 1$.} \\
&= \sqrt{\mathbf{w}^\top \mathbf{w}} && \text{by definition of $\|\mathbf{w}\|$} \\
&\le \sqrt{M} &&\text{By (2)} \\
& \textrm{ }\\
&\Rightarrow M\gamma \le \sqrt{M} \\
&\Rightarrow M^2\gamma^2 \le M \\
&\Rightarrow M \le \frac{1}{\gamma^2} && \text{And hence, the number of updates $M$ is bounded from above by a constant.}
\end{align}
Quiz: Given the theorem above, what can you say about the margin of a classifier (what is more desirable, a large margin or a small margin?) Can you characterize data sets for which the Perceptron algorithm will converge quickly? Draw an example.
