Stochastic Gradient Descent
Summary
This is an optimization method, used in many algorithms such as Support Vector Machines or Conditional Random Fields to efficiently optimize the objective function, especially in the online setting. Stochastic optimizations like this method are known to be faster when trained with large, redundant data sets.
Gradient Descent
Given a function to be minimized and a point , let then we can say
for some small enough . Using this inequality, we can get a (local) minimum of the objective function using the following steps:
- Initialize
- Repeat the step above until the objective function converges to a local minimum
Stochastic Gradient Descent
One of the problems of the gradient descent method above is that calculating the gradient could be an expensive computation depending on the objective function or the size of the data set. Suppose your objective function is . If the objective function can be decomposed as the following,
where indicates the -th example(sometimes is a batch instead of one example), we can make the process stochastic. To make each step computationally efficient, a subset of the summand function is sampled. The procedure can be described as the following pseudocode:
- Initialize
- Repeat until convergence
- Sample examples
- For each example sampled
where is the learning rate. Note that this method is no different from the plain gradient descent method when the batch size becomes the number of examples. For computational efficiency, small batch size around 5~20 turn out to be most efficient.
Pros
When this method is used for very large data sets that has redundant information among examples, it is much faster than the plain gradient descent because it requires less computation each iteration. Also, it is known to be better with noisy data since it samples example to compute gradient.
Cons
The convergence rate is slower than second-order gradient methods. However the speedup coming from computationally efficient iterations are usually greater and the method can converge faster if learning rate is adjusted as the procedure goes on. Also it tends to keep bouncing around the minimum unless the learning rate is reduced in the later iterations.