Expectation Regularization
This is a method introduced in G.S Mann and A. McCallum, ICML 2007. It is often served as a regularized term with the likelihood function. In practice human often have an insight of label prior distribution. This method introduced a way to take advantage of this prior knowledge.
Let's denote human-provided prior as and empirical label distribution as . The empirical label distribution is computed over unlabeled data set ,
We want to minimize the distance between and , denoted as . KL-distance is used here so the regularization becomes
For semi-supervised learning purposes, we can augment the objective function by adding regularization term. For example, the new conditional likelihood of data becomes
where is the labeled data set.
Note that this is a global regularizer instead of a local one, in which case it would assign all instances to the majority of the class.