Entropy Minimization for Semi-supervised Learning

Minimum entropy regularization can be applied to any model of posterior distribution.

The learning set is denoted $L_{n}=\{x_{i},z_{i}\}_{i=1}^{n}$, where $z_{i}\in \{0,1\}^{K}$: If $x_{i}$ is labeled as $w_{i}$, then $z_{ik}=1$ and $z_{il}=0$ for $l\not =k$; if $x_{i}$ is unlabeled, then $z_{il}=1$ for $l=1\dots K$.

The conditional entropy of class labels conditioned on the observed variables:

$H(Y|X,Z;L_{n})=-{\frac {1}{n}}\sum _{i=1}^{n}\sum _{k=1}^{K}P(Y=w_{k}|x_{i},z_{i}){\text{log}}P(Y=w_{k}|x_{i},z_{i})$

The posterior distribution is defined as

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle C(\theta, \lambda; L_{n}) = L(\theta; \mathcal{L}_{n}) - \lambda H(Y|X,Z; \mathcal{L}_{n}) \\ = \sum^{n}_{i=1} \text{log}(\sum^{K}_{k=1} z_{ik}P(Y^{i}=w_{k}|X^{i})) }