Expectation Regularization

This is a method introduced in G.S Mann and A. McCallum, ICML 2007. It is often served as a regularized term with the likelihood function. In practice human often have an insight of label prior distribution. This method introduced a way to take advantage of this prior knowledge.

Let's denote human-provided prior as ${\tilde {p}}$ and empirical label distribution as ${\hat {p}}$. The empirical label distribution is computed over unlabeled data set $U$,

${\hat {p}}_{\theta }(y)={\frac {\sum _{x\in U}p_{\theta }(y|x)}{|U|}}$

We want to minimize the distance between ${\tilde {p}}$ and ${\hat {p}}$, denoted as $\triangle ({\hat {p}},{\tilde {p}})$. KL-distance is used here so the regularization becomes

$D({\tilde {p}}||{\hat {p}})=\sum _{y}{\tilde {p}}(y){\text{log}}{\frac {{\tilde {p}}(y)}{{\hat {p}}(y)}}=H({\tilde {p}},{\hat {p}})-H({\tilde {p}})$

For semi-supervised learning purposes, we can augment the objective function by adding regularization term. For example, the new conditional likelihood of data becomes

$l(\theta ;D,U)=\sum _{n}{\text{log}}p_{\theta }(y^{(n)}|x^{(n)})-\lambda \triangle ({\tilde {p}},{\hat {p}})$

Note that this is a global regularizer instead of a local one, in which case it would assign all instances to the majority of the class.