Entropy Minimization for Semi-supervised Learning

This is a method introduced in Y. Grandvalet.

Minimum entropy regularization can be applied to any model of posterior distribution. For this technique, one assumption for unlabeled examples to be informative is that classes are well apart, separated by a low density area.

The learning set is denoted ${\mathcal {L}}_{n}=\{X^{(i)},Z^{(i)}\}_{i=1}^{n}$, where $Z^{(i)}\in \{0,1\}^{K}$: If $X^{(i)}$ is labeled as $w_{i}$, then $Z_{k}^{(i)}=1$ and $Z_{l}^{(i)}=0$ for $l\not =k$; if $X^{(i)}$ is unlabeled, then $Z_{l}^{(i)}=1$ for $l=1\dots K$.

The conditional entropy of class labels conditioned on the observed variables:

$H(Y|X,Z;{\mathcal {L}}_{n})=-{\frac {1}{n}}\sum _{i=1}^{n}\sum _{k=1}^{K}P(Y^{(i)}=w_{k}|X^{(i)},Z^{(i)}){\text{log}}P(Y^{(i)}=w_{k}|X^{(i)},Z^{(i)})$

Assuming that labels are missing at random, we have that

$P(Y^{(i)}=w_{k}|X^{(i)},Z^{(i)})={\frac {Z_{k}^{(i)}P(Y^{(i)}=w_{k}|X^{(i)})}{\sum _{k=1}^{K}Z_{l}^{(i)}P(Y^{(i)}=w_{k}|X^{(i)})}}$

The posterior distribution is defined as the conditional log likelihood and an entropy-regularized term:

${\begin{alignedat}{2}C({\boldsymbol {\theta }},\lambda ;{\mathcal {L}}_{n})&=L({\boldsymbol {\theta }};{\mathcal {L}}_{n})-\lambda H(Y|X,Z;{\mathcal {L}}_{n})\\&=\sum _{i=1}^{n}{\text{log}}(\sum _{k=1}^{K}Z_{k}^{(i)}P(Y^{(i)}=w_{k}|X^{(i)}))+\lambda \sum _{i=1}^{n}\sum _{k=1}^{K}P(Y^{(i)}=w_{k}|X^{(i)},Z^{(i)}){\text{log}}P(Y^{i}=w_{k}|X^{i},Z^{i})\end{alignedat}}$

Minimum entropy regularizers have been used to encode learnability of priors M. Brand and to learn weight function parameters in the context of transduction in manifold learning Zhu et al..