This Method is used to sample from the stable joint distribution of two or more random variables when accurate computation of the integral or a marginal is intractable. Usually some variables in this set of random variables are the actual observables and hence there values need not be sampled in the iterations. This form of approximate inference method is generally used when doing posterior probability inference in probabilistic graphical models where computation of marginals are intractable.
Gibbs sampling was introduced in the context of image processing by Geman and Geman. The Gibbs sampler is a technique for generating random variables from a (marginal) distribution indirectly, without having to calculate the density. Thus, if we are given with conditional densities , we can use Gibbs sampling to calculate the marginal distributions or any other function of .
1. Take some initial values
2. Repeat for :
3. Continue step 2 until joint distribution of doesn't change.
Under regularity conditions, it can be shown that this procedure eventually stabilizes, and the resulting random variables are indeed a sample from
A Simple proof for bivariate case
Consider a bivariate system of bernoulli random variables and . Define two matrices and such that and . Then the transition probability of to can be given as . If the initial distribution to start with was , then at the iteration, . It is well known that as , approaches a stationary point. The stationary point represents .
Gibbs sampler requires certain number of iterations until it approaches the stationary state, and generate samples from the marginal distribution. To allow this, first few samples (typically in the order of 500-1000) are discarded. This is known as burnout.
For detecting if the sampling has approached the stationary state (the burnout parameter), various strategies have been proposed. Gelfand and Smith, 1990 suggest monitoring density estimates from independent generated samples, and choosing the first point in time when these densities appear to be the same. Another strategy is to monitor a sequence of weights that measure by how much the sampled and desired distribution differ. However, none of these strategies apply to all kinds of distributions.
Similarity with other methods
Gibbs sampling is a special case of Metropolis-Hastings algorithm where the proposal distribution is the conditional distribution and every sample is accepted.
Gibbs sample can be observed as a variant of the EM algorithm for exponential models. The required connection is to view the latent variables as parameters to the Gibbs sampler, and instead of maximizing; sample from the conditional distribution.
1. Geman and Geman
3. Trevor Hastie, Robert Tibshirani, Jerome Friedman. The Elements of Statistical Learning.