Linear Regression

Old Recorded Lectures

Assumptions

In words, we assume that the data is drawn from a "line" ${\mathbf{w}}^T\mathbf{x}$ through the origin (one can always add a bias / offset through an additional dimension, similar to the Perceptron). For each data point with features ${\mathbf{x}}_i$, the label $y$ is drawn from a Gaussian with mean ${\mathbf{w}}^T{\mathbf{x}}_i$ and variance ${\sigma }^2$. Our task is to estimate the slope $\mathbf{w}$ from the data.

Estimating with MLE

We are minimizing a loss function, $l(\mathbf{w})=\frac{1}{n}\sum _{i=1}^n({\mathbf{x}}_i^T\mathbf{w}-y_i{)}^2$. This particular loss function is also known as the squared loss or Ordinary Least Squares (OLS). In this form, it has a natural interpretation as the average squared error of the prediction over the training set. OLS can be optimized with gradient descent, Newton's method, or in closed form.

Estimating with MAP

This objective is known as Ridge Regression. It has a closed form solution of: $\hat{\mathbf{w}}=(\mathbf{X}{\mathbf{X}}^T+\lambda \mathbf{I}{)}^{-1}\mathbf{X}{\mathbf{y}}^T,$ where $\mathbf{X}=[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n]$ and $\mathbf{y}=[y_1,\dots ,y_n]$. The solution must always exist and be unique (why?).

Summary

• ${\min }_{\mathbf{w}}\frac{1}{n}\sum _{i=1}^n({\mathbf{x}}_i^T\mathbf{w}-y_i{)}^2$.
• Squared loss.
• No regularization.
• Closed form: $\mathbf{w}=(\mathbf{X}{\mathbf{X}}^{\mathbf{T}}{)}^{-1}\mathbf{X}{\mathbf{y}}^T$.

• ${\min }_{\mathbf{w}}\frac{1}{n}\sum _{i=1}^n({\mathbf{x}}_i^T\mathbf{w}-y_i{)}^2+\lambda ||\mathbf{w}|{|}_2^2$.
• Squared loss.
• $l2\text{-regularization}$.
• Closed form: $\mathbf{w}=(\mathbf{X}{\mathbf{X}}^{\mathbf{T}}+\lambda \mathbf{I}{)}^{-1}\mathbf{X}{\mathbf{y}}^T$.