Stochastic Gradient Descent

From Cohen Courses
Jump to navigationJump to search


This is an optimization method, used in many algorithms such as Support Vector Machines or Conditional Random Fields to efficiently optimize the objective function, especially in the online setting. Stochastic optimizations like this method are known to be faster when trained with large, redundant data sets.

Gradient Descent

Given a function to be minimized and a point , let then we can say

for some small enough . Using this inequality, we can get a (local) minimum of the objective function using the following steps:

  • Initialize
  • Repeat the step above until the objective function converges to a local minimum

Stochastic Gradient Descent

One of the problems of the gradient descent method above is that calculating the gradient could be an expensive computation depending on the objective function or the size of the data set. Suppose your objective function is . If the objective function can be decomposed as the following,

where indicates the -th example(sometimes is a batch instead of one example), we can make the process stochastic. To make each step computationally efficient, a subset of the summand function is sampled. The procedure can be described as the following pseudocode:

  • Initialize
  • Repeat until convergence
    • Sample examples
    • For each example sampled

where is the learning rate. Note that this method is no different from the plain gradient descent method when the batch size becomes the number of examples. For computational efficiency, small batch size around 5~20 turn out to be most efficient.


When this method is used for very large data sets that has redundant information among examples, it is much faster than the plain gradient descent because it requires less computation each iteration. Also, it is known to be better with noisy data since it samples example to compute gradient.


The convergence rate is slower than second-order gradient methods. However the speedup coming from computationally efficient iterations are usually greater and the method can converge faster if learning rate is adjusted as the procedure goes on. Also it tends to keep bouncing around the minimum unless the learning rate is reduced in the later iterations.

Related Papers