Kernels

Handcrafted Feature Expansion

Advantage: It is simple, and your problem stays convex and well behaved. (i.e. you can still use your original gradient descent code, just with the higher dimensional representation)

Disadvantage: $\varphi (\mathbf{x})$ might be very high dimensional.

Consider the following example: $\mathbf{x}=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_d\end{array}\right)$, and define $\varphi (\mathbf{x})=\left(\begin{array}{c}1\\ x_1\\ ⋮\\ x_d\\ x_1{x}_2\\ ⋮\\ x_{d-1}{x}_d\\ ⋮\\ x_1{x}_2\cdots x_d\end{array}\right)$.

Quiz: What is the dimensionality of $\varphi (\mathbf{x})$?

This new representation, $\varphi (\mathbf{x})$, is very expressive and allows for complicated non-linear decision boundaries - but the dimensionality is extremely high. This makes our algorithm unbearable (and quickly prohibitively) slow.

The Kernel Trick

Gradient Descent with Squared Loss

The kernel trick is a way to get around this dilemma by learning a function in the much higher dimensional space, without ever computing a single vector $\varphi (\mathbf{x})$ or ever computing the full vector $\mathbf{w}$. It is a little magical.

It is based on the following observation: If we use gradient descent with any one of our standard loss functions, the gradient is a linear combination of the input samples. For example, let us take a look at the squared loss:

Formally, the argument is by induction. $\mathbf{w}$ is trivially a linear combination of our training vectors for ${\mathbf{w}}_0$ (base case). If we apply the inductive hypothesis for ${\mathbf{w}}_t$ it follows for ${\mathbf{w}}_{t+1}$.

Inner-Product Computation

Let's go back to the previous example, $\varphi (\mathbf{x})=\left(\begin{array}{c}1\\ x_1\\ ⋮\\ x_d\\ x_1{x}_2\\ ⋮\\ x_{d-1}{x}_d\\ ⋮\\ x_1{x}_2\cdots x_d\end{array}\right)$.

The inner product $\varphi (\mathbf{x}{)}^{\mathrm{\top }}\varphi (\mathbf{z})$ can be formulated as: $$\varphi (\mathbf{x}{)}^{\mathrm{\top }}\varphi (\mathbf{z})=1\cdot 1+x_1{z}_1+x_2{z}_2+\cdots +x_1{x}_2{z}_1{z}_2+\cdots +x_1\cdots x_d{z}_1\cdots z_d=\prod _{k=1}^d(1+x_k{z}_k)\text{.}$$ The sum of $2^d$ terms becomes the product of $d$ terms. We can compute the inner-product from the above formula in time $O(d)$ instead of $O(2^d)$! We define the function $$\underset{\text{this is called the}\mathbf{\text{ kernel function}}}{\underbrace{\mathsf{k}({\mathbf{x}}_i,{\mathbf{x}}_j)}}=\varphi ({\mathbf{x}}_i{)}^{\mathrm{\top }}\varphi ({\mathbf{x}}_j).$$ With a finite training set of $n$ samples, inner products are often pre-computed and stored in a Kernel Matrix: $${\mathsf{K}}_{ij}=\varphi ({\mathbf{x}}_i{)}^{\mathrm{\top }}\varphi ({\mathbf{x}}_j).$$ If we store the matrix $\mathsf{K}$, we only need to do simple inner-product look-ups and low-dimensional computations throughout the gradient descent algorithm. The final classifier becomes: $$h({\mathbf{x}}_t)=\sum _{j=1}^n{\alpha }_j\mathsf{k}({\mathbf{x}}_j,{\mathbf{x}}_t).$$

During training in the new high dimensional space of $\varphi (\mathbf{x})$ we want to compute ${\gamma }_i$ through kernels, without ever computing any $\varphi ({\mathbf{x}}_i)$ or even $\mathbf{w}$. We previously established that $\mathbf{w}=\sum _{j=1}^n{\alpha }_j\varphi ({\mathbf{x}}_j)$, and ${\gamma }_i=2({\mathbf{w}}^{\mathrm{\top }}\varphi ({\mathbf{x}}_i)-y_i)$. It follows that ${\gamma }_i=2(\sum _{j=1}^n{\alpha }_j{K}_{ij})-y_i)$. The gradient update in iteration $t+1$ becomes $${\alpha }_i^{t+1}\leftarrow {\alpha }_i^t-2s(\sum _{j=1}^n{\alpha }_j^t{K}_{ij})-y_i).$$ As we have $n$ such updates to do, the amount of work per gradient update in the transformed space is $O(n^2)$ --- far better than $O(2^d)$.

General Kernels

Linear: $\mathsf{K}(\mathbf{x},\mathbf{z})={\mathbf{x}}^{\mathrm{\top }}\mathbf{z}$.

(The linear kernel is equivalent to just using a good old linear classifier - but it can be faster to use a kernel matrix if the dimensionality $d$ of the data is high.)

Polynomial: $\mathsf{K}(\mathbf{x},\mathbf{z})=(1+{\mathbf{x}}^{\mathrm{\top }}\mathbf{z}{)}^d$.

Radial Basis Function (RBF) (aka Gaussian Kernel): $\mathsf{K}(\mathbf{x},\mathbf{z})=e^{\frac{-\parallel \mathbf{x}-\mathbf{z}{\parallel }^2}{{\sigma }^2}}$.

The RBF kernel is the most popular Kernel! It is a Universal approximator!! Its corresponding feature vector is infinite dimensional and cannot be computed. However, very effective low dimensional approximations exist (see this paper).

Exponential Kernel: $\mathsf{K}(\mathbf{x},\mathbf{z})=e^{\frac{-\parallel \mathbf{x}-\mathbf{z}\parallel }{2{\sigma }^2}}$

Laplacian Kernel: $\mathsf{K}(\mathbf{x},\mathbf{z})=e^{\frac{-|\mathbf{x}-\mathbf{z}|}{\sigma }}$

Sigmoid Kernel: $\mathsf{K}(\mathbf{x},\mathbf{z})=\tanh (\mathbf{a}{\mathbf{x}}^{\mathrm{\top }}+c)$

Kernel functions

Can any function $\mathsf{K}(\cdot ,\cdot )\to \mathcal{R}$ be used as a kernel?

Remember ${\mathsf{K}}_{ij}=\varphi ({\mathbf{x}}_i{)}^{\mathrm{\top }}\varphi ({\mathbf{x}}_j)$. So $\mathsf{K}={\mathrm{\Phi }}^{\mathrm{\top }}\mathrm{\Phi }$, where $\mathrm{\Phi }=[\varphi ({\mathbf{x}}_1),\dots ,\varphi ({\mathbf{x}}_n)]$. It follows that $\mathsf{K}$ is p.s.d., because ${\mathbf{q}}^{\mathrm{\top }}\mathsf{K}\mathbf{q}=({\mathrm{\Phi }}^{\mathrm{\top }}\mathbf{q}{)}^2\ge 0$. Inversely, if any matrix $\mathbf{A}$ is p.s.d., it can be decomposed as $A={\mathrm{\Phi }}^{\mathrm{\top }}\mathrm{\Phi }$ for some realization of $\mathrm{\Phi }$.

You can even define kernels over sets, strings, graphs and molecules.

Figure 1: The demo shows how kernel function solves the problem linear classifiers can not solve. RBF works well with the decision boundary in this case.