Posterior Regularization for Expectation Maximization

Summary

This is a method to impose contraints on posteriors in the Expectation Maximization algorithm, allowing a finer-level control over these posteriors.

Method Description

For a given set x of observed data, a set of latent data z and a set of parameters $\theta$, the Expectation Maximization algorithm can be viewed as the alternation between two maximization steps of the function $F(q,\theta )$.

The E-step is defined as:

$q^{t+1}=argmax_{q}F(q,\theta ^{t})=argmax_{q}[-D_{KL}(q||p_{\theta ^{t}}(z|x))]$

where $D_{KL}$ is the Kullback-Leibler divergence given by $D_{KL}(q||p)=E_{q}[log{\frac {q}{p}}]$

The M-step is defined as:

$\theta ^{t+1}=$