Expectation Regularization

This is a method introduced in G.S Mann and A. McCallum, ICML 2007. It is often served as a regularized term with the likelihood function. In practice human often have an insight of label prior distribution. This method introduced a way to take advantage of this prior knowledge.

Let's denote human-provided prior as ${\tilde {p}}$. We minimizes the distance between ${\tilde {p}}$ and ${\hat {p}}$. KL-distance is used here so the regularization becomes

$D({\tilde {p}}||{\hat {p}})=\sum _{y}{\tilde {p}}(y){\text{log}}{\frac {{\tilde {p}}(y)}{{\hat {p}}(y)}}=H({\tilde {p}},{\hat {p}})-H({\tilde {p}})$

For semi-supervised learning purposes, we can augment the objective function by adding regularization term. For example, the new conditional likelihood of data becomes

$l(\theta ;D,U)=\sum _{n}{\text{log}}p_{\theta }(y^{(n)}|x^{(n)})-\lambda \triangle ({\tilde {p}},{\hat {p}})$