Entropy Minimization for Semi-supervised Learning

Minimum entropy regularization can be applied to any model of posterior distribution.

The learning set is denoted ${\mathcal {L}}_{n}=\{X^{(i)},Z^{(i)}\}_{i=1}^{n}$, where $z_{i}\in \{0,1\}^{K}$: If $x_{i}$ is labeled as $w_{i}$, then $z_{ik}=1$ and $z_{il}=0$ for $l\not =k$; if $x_{i}$ is unlabeled, then $z_{il}=1$ for $l=1\dots K$.

The conditional entropy of class labels conditioned on the observed variables:

$H(Y|X,Z;L_{n})=-{\frac {1}{n}}\sum _{i=1}^{n}\sum _{k=1}^{K}P(Y=w_{k}|x_{i},z_{i}){\text{log}}P(Y=w_{k}|x_{i},z_{i})$

The posterior distribution is defined as

${\begin{alignedat}{2}C({\boldsymbol {\theta }},\lambda ;L_{n})&=L({\boldsymbol {\theta }};{\mathcal {L}}_{n})-\lambda H(Y|X,Z;{\mathcal {L}}_{n})\\&=\sum _{i=1}^{n}{\text{log}}(\sum _{k=1}^{K}Z_{ik}P(Y^{i}=w_{k}|X^{i}))+\lambda \sum _{i=1}^{n}\sum _{k=1}^{K}P(Y^{i}=w_{k}|X^{i},Z^{i}){\text{log}}P(Y^{i}=w_{k}|X^{i},Z^{i})\end{alignedat}}$