Support Vector Machines

This is a machine learning model.

Summary

Support Vector Machines are supervised models that take positive and negative instances of classes in some feature space as training instances, and learn a linear decision boundary between the two classes. Unlike probability based methods such as logistic regression that attempt to learn a hyperplane that maximizes the likelihood of the training data, SVMs rely on margin and learn hyperplanes that maximize the distance separation between positive and negative instances. For cases where the training data isn't perfectly separable, they make use of slack variables.

Formal Description

We are given some training data ${\mathcal {D}}$, a set of n points of the form

${\mathcal {D}}=\left\{(\mathbf {x} _{i},y_{i})\mid \mathbf {x} _{i}\in \mathbb {R} ^{p},\,y_{i}\in \{-1,1\}\right\}_{i=1}^{n}$

where the y_i is either 1 or −1, indicating the class to which the point $\mathbf {x} _{i}$ belongs. Each $\mathbf {x} _{i}$ is a p-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y_{i}=1$ from those having $y_{i}=-1$.

We want to choose the ${\mathbf {w} }$ and $b$ to maximize the margin, or distance between the parallel hyperplanes that are as far apart as possible while still separating the data, i.e.

Minimize (in ${\mathbf {w} ,b}$)

$\|\mathbf {w} \|$

subject to (for any $i=1,\dots ,n$)

$y_{i}(\mathbf {w} \cdot \mathbf {x_{i}} -b)\geq 1.\,$

We can substitute ||w|| with ${\tfrac {1}{2}}\|\mathbf {w} \|^{2}$ without changing the solution. This now becomes a quadratic programming problem.

Training

There exist several specialized algorithms for quickly solving the Quadratic Programming problem that arises from SVMs, mostly reliant on heuristics for breaking the problem down into smaller, more-manageable chunks. A common method is the Sequential Minimal Optimization (SMO) algorithm, which breaks the problem down into 2-dimensional sub-problems that can be solved analytically and runs in cubic time.

Advantages

• They are more robust to overfitting, since the final model relies only on support vectors, as opposed to all the instances of the training set.
• The use of the Kernel trick allows us to efficiently use custom kernels for any given task
• They can easily handle very high dimensional data

Applications

They are applicable for a variety of binary as well as multiclass classification problems such as text classification, and have recently also been applied for regression problems.