Kernels continued

Well-defined kernels

• Linear: $\mathsf{k}(\mathbf{x},\mathbf{z})={\mathbf{x}}^{\mathrm{\top }}\mathbf{z}$
• RBF: $\mathsf{k}(\mathbf{x},\mathbf{z})=e^{-\frac{(\mathbf{x}-\mathbf{z}{)}^2}{{\sigma }^2}}$
• Polynomial: $\mathsf{k}(\mathbf{x},\mathbf{z})=(1+{\mathbf{x}}^{\mathrm{\top }}\mathbf{z}{)}^d$

1. $\mathsf{k}(\mathbf{x},\mathbf{z})={\mathbf{x}}^{\mathrm{\top }}\mathbf{z}$
2. $\mathsf{k}(\mathbf{x},\mathbf{z})=c{\mathsf{k}}_{\mathsf{1}}(\mathbf{x},\mathbf{z})$
3. $\mathsf{k}(\mathbf{x},\mathbf{z})={\mathsf{k}}_{\mathsf{1}}(\mathbf{x},\mathbf{z})+{\mathsf{k}}_{\mathsf{2}}(\mathbf{x},\mathbf{z})$
4. $\mathsf{k}(\mathbf{x},\mathbf{z})=g(\mathsf{k}(\mathbf{x},\mathbf{z}))$
5. $\mathsf{k}(\mathbf{x},\mathbf{z})={\mathsf{k}}_{\mathsf{1}}(\mathbf{x},\mathbf{z}){\mathsf{k}}_{\mathsf{2}}(\mathbf{x},\mathbf{z})$
6. $\mathsf{k}(\mathbf{x},\mathbf{z})=f(\mathbf{x}){\mathsf{k}}_{\mathsf{1}}(\mathbf{x},\mathbf{z})f(\mathbf{z})$
7. $\mathsf{k}(\mathbf{x},\mathbf{z})=e^{{\mathsf{k}}_{\mathsf{1}}(\mathbf{x},\mathbf{z})}$
8. $\mathsf{k}(\mathbf{x},\mathbf{z})={\mathbf{x}}^{\mathrm{\top }}\mathbf{A}\mathbf{z}$

1. All eigenvalues of $K$ are non-negative.
2. $\mathrm{\exists }\text{ real matrix }P\text{ s.t. }K=P^{\mathrm{\top }}P.$
3. $\mathrm{\forall }\text{ real vector }\mathbf{x},{\mathbf{x}}^{\mathrm{\top }}K\mathbf{x}\ge 0.$

You can even define kernels of sets, or strings or molecules.

Kernel Machines

(In practice) an algorithm can be kernelized in 2 steps:

1. Prove that the solution lies in the span of the training points (i.e. $\mathbf{w}=\sum _{i=1}^n{\alpha }_i{\mathbf{x}}_i$ for some ${\alpha }_i$)
2. Rewrite the algorithm and the classifier so that all training or testing inputs ${\mathbf{x}}_i$ are only accessed in inner-products with other inputs, e.g. ${\mathbf{x}}_i^{\mathrm{\top }}{\mathbf{x}}_j$.
3. Define a kernel function and substitute $\mathsf{k}({\mathbf{x}}_i,{\mathbf{x}}_j)$ for ${\mathbf{x}}_i^{\mathrm{\top }}{\mathbf{x}}_j$.

Kernelized Linear Regression

Recap

Vanilla Ordinary Least Squares Regression (OLS) [also referred to as linear regression] minimizes the following squared loss regression loss function, $$\underset{\mathbf{w}}{min}\sum _{i=1}^n({\mathbf{w}}^{\mathrm{\top }}{\mathbf{x}}_i-y_i{)}^2,$$ to find the hyper-plane $\mathbf{w}$. The prediction at a test-point is simply $h(\mathbf{x})={\mathbf{w}}^{\mathrm{\top }}\mathbf{x}$.
If we let $\mathbf{X}=[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n]$ and $\mathbf{y}=[y_1,\dots ,y_n{]}^{\mathrm{\top }}$, the solution of OLS can be written in closed form: $$\mathbf{w}=(\mathbf{X}{\mathbf{X}}^{\mathrm{\top }}{)}^{-1}\mathbf{X}\mathbf{y}(5)$$

Kernelization

We begin by expressing the solution $\mathbf{w}$ as a linear combination of the training inputs $$\mathbf{w}=\sum _{i=1}^n{\alpha }_i{\mathbf{x}}_i=\mathbf{X}\overrightarrow{\alpha }.$$ We derived in the previous lecture that such a vector $\overrightarrow{\alpha }$ must always exist by observing the gradient updates that occur if (5) is minimized with gradient descent and the initial vector is set to ${\mathbf{w}}_0=\overrightarrow{0}$ (because the squared loss is convex the solution is independent of its initialization.)

Similarly, during testing a test point is only accessed through inner-products with training inputs: $$h(\mathbf{z})={\mathbf{w}}^{\mathrm{\top }}\mathbf{z}=\sum _{i=1}^n{\alpha }_i{\mathbf{x}}_i^{\mathrm{\top }}\mathbf{z}.$$ We can now immediately kernelize the algorithm by substituting $k(\mathbf{x},\mathbf{z})$ for any inner-product ${\mathbf{x}}^{\mathrm{\top }}\mathbf{z}$. It remains to show that we can also solve for the values of $\alpha$ in closed form. As it turns out, this is straight-forward.

Testing

Remember that we defined $\mathbf{w}=\mathbf{X}\overrightarrow{\alpha }.$ The prediction of a test point $\mathbf{z}$ then becomes $$h(\mathbf{z})={\mathbf{z}}^{\mathrm{\top }}\mathbf{w}={\mathbf{z}}^{\mathrm{\top }}\underset{\mathbf{w}}{\underbrace{\mathbf{X}\overrightarrow{\alpha }}}=\underset{{\mathbf{z}}^{\mathrm{\top }}\mathbf{X}}{\underbrace{{\mathbf{k}}_{\ast }}}\underset{\overrightarrow{\alpha }}{\underbrace{(\mathbf{K}+{\tau }^2\mathbf{I}{)}^{-1}\mathbf{y}}}={\mathbf{k}}_{\ast }\overrightarrow{\alpha },$$ or, if everything is in closed form: $$h(\mathbf{z})={\mathbf{k}}_{\ast }(\mathbf{K}+{\tau }^2\mathbf{I}{)}^{-1}\mathbf{y},$$ where ${\mathbf{k}}_{\ast }$ is the kernel (vector) of the test point with the training points, i.e. the $i^{th}$ dimension corresponds to $[{\mathbf{k}}_{\ast }{]}_i=\varphi (\mathbf{z}{)}^{\mathrm{\top }}\varphi ({\mathbf{x}}_i)$, the inner-product between the test point $\mathbf{z}$ with the training point ${\mathbf{x}}_i$ after the mapping into feature space through $\varphi$.

Nearest Neighbors

Quiz 3: Let $D=\{({\mathbf{x}}_1,y_1),\dots ,({\mathbf{x}}_n,y_n)\}.$ How can you kernelize nearest neighbors (with Euclidean distances)?

Kernel SVM

The original, primal SVM is a quadratic programming problem: $$\begin{array}{rl}& \underset{\mathbf{w},b}{min}{\mathbf{w}}^{\mathrm{\top }}\mathbf{w}+C\sum _{i=1}^n{\xi }_i\\ \text{s.t. }\mathrm{\forall }i,& y_i({\mathbf{w}}^{\mathrm{\top }}{\mathbf{x}}_i+b)\ge 1-{\xi }_i\\ & {\xi }_i\ge 0\end{array}$$ has the dual form $$\begin{array}{rl}& \underset{{\alpha }_1,\cdots ,{\alpha }_n}{min}\frac{1}{2}\sum _{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{K}_{ij}-\sum _{i=1}^n{\alpha }_i\\ \text{s.t.}& 0\le {\alpha }_i\le C\\ & \sum _{i=1}^n{\alpha }_i{y}_i=0\end{array}$$
where $\mathbf{w}=\sum _{i=1}^n{\alpha }_i{y}_i\varphi ({\mathbf{x}}_i)$ (although this is never computed) and $$h(\mathbf{x})=\text{sign}(\sum _{i=1}^n{\alpha }_i{y}_{i}k({\mathbf{x}}_i,\mathbf{x})+b).$$

Support Vectors

Recovering $b$

One apparent problem with the dual version is that $b$ is no longer part of the optimization. However, we need it to perform classification. Luckily, we know that the primal solution and the dual solution are identical. In the duel, support vectors are those with ${\alpha }_i>0$. We can then solve the following equation for $b$ $$\begin{array}{rl}{y}_i({\mathbf{w}}^{\mathrm{\top }}\varphi (x_i)+b)& =1\\ y_i(\sum _j{y}_j{\alpha }_{j}k({\mathbf{x}}_j,{\mathbf{x}}_i)+b)& =1\\ y_i-\sum _j{y}_j{\alpha }_{j}k({\mathbf{x}}_j,{\mathbf{x}}_i)& =b\end{array}$$ This allows us to solve for $b$ from the support vectors (in practice it is best to average the $b$ from several support vectors, as there may be numerical precision problems).

Kernel SVM - the smart nearest neighbor

Do you remember the k-nearest neighbor algorithm? For binary classification problems ($y_i\in \{+1,-1\}$), we can write the decision function for a test point $\mathbf{z}$ as $$h(\mathbf{z})=\text{sign}(\sum _{i=1}^n{y}_i{\delta }^{nn}({\mathbf{x}}_i,\mathbf{z})),$$ where ${\delta }^{nn}(\mathbf{z},{\mathbf{x}}_i)\in \{0,1\}$ with ${\delta }^{nn}(\mathbf{z},{\mathbf{x}}_i)=1$ only if ${\mathbf{x}}_i$ is one of the $k$ nearest neighbors of test point $\mathbf{z}$. The SVM decision function $$h(\mathbf{z})=\text{sign}(\sum _{i=1}^n{y}_i{\alpha }_{i}k({\mathbf{x}}_i,\mathbf{z})+b)$$ is very similar, but instead of limiting the decision to the $k$ nearest neighbors, it considers all training points but the kernel function assigns more weight to those that are closer (large $k(\mathbf{z},{\mathbf{x}}_i))$. In some sense you can view the RBF kernel as a soft nearest neighbor assignment, as the exponential decay with distance will assign almost no weight to all but the neighboring points of $\mathbf{z}$. The Kernel SVM algorithm also learns a weight ${\alpha }_i>0$ for each training point and a bias $b$ and it essentially "removes" useless training points by setting many ${\alpha }_i=0$.