Posterior Regularization for Expectation Maximization

Summary

This is a method to impose contraints on posteriors in the Expectation Maximization algorithm, allowing a finer-level control over these posteriors.

Method Description

For a given set $x\in X$ of observed data, a set of latent data $z\in Z$ and a set of parameters $\theta$, the Expectation Maximization algorithm can be viewed as the alternation between two maximization steps of the function $F(q,\theta )$, by marginalizing different free variables.

The E-step is defined as:

$q^{t+1}=argmax_{q}F(q,\theta ^{t})=argmax_{q}[-D_{KL}(q||p_{\theta ^{t}}(z|x))]$

where $D_{KL}$ is the Kullback-Leibler divergence given by $D_{KL}(q||p)=E_{q}[log{\frac {q}{p}}]$, and q(z|x) is an arbitrary probability distribution over the latent variable z.

The M-step is defined as:

$\theta ^{t+1}=argmax_{\theta }F(q^{t+1},\theta )=argmax_{\theta }\ E_{X}[\sum _{s}q^{t+1}(z|x)\ log\ p_{\theta }(x,z)]$

The goal of this method is to define a way to constrains over posteriors.