Entropy Gradient for Semi-Supervised Conditional Random Fields

This method is used by Mann and McCallum, 2007 for efficient computation of the entropy gradient used as a regularizer to train semi-supervised conditional random fields. The method is an improvement over the original proposed approach by Jiao et al. 2006 in terms of computing the gradient on unlabeled part of the training data.

Summary

Entropy regularization (ER) is a method applied to semi-supervised learning that augments a standard conditional likelihood objective function with an additional term that aims to minimize the predicted label entropy on unlabeled data. By insisting on peaked, confident predictions, ER guides the decision boundary away from dense regions of input space. Entropy regularization for semi-supervised learning was first proposed for classification tasks by Grandvalet and Bengio, 2004.

Motivation

Jiao et al. 2006 apply this method to linear chain CRFs and demonstrate encouraging accuracy improvements on a gene-name-tagging task. However, the method they presented for calculating the gradient of the entropy takes substantially greater time than the raditional supervised-only gradient. Whereas supervised training requires only classic forward-backward style algorithms, taking time $O(ns^{2})$ (sequence length times the square of the number of labels), their training method takes $O(n^{2}s^{3})$ — a factor of $O(ns)$ more.

This method proposed in Mann and McCallum, 2007 introduces a more efficient way to derive entropy gradient based on dynamic programming that has the same asymptotic time complexity as that of a supervised CRF training process, $O(ns2)$. This calculation introduces the concept of subsequence constrained entropy — the entropy of a CRF for an observed data sequence when part of the label sequence is fixed. This method is especially useful for training CRFs on larger unannotated data sets.

Semi-Supervised CRF Training

A standard linear chain CRF is trained by maximizing the log-likelihood $L(\theta ;D)$ on a labeled dataset $D$. Gradient methods like L-BFGS are commonly used to optimize the following objective function:

For semi-supervised training by entropy regularization, the objective function is augmented by adding the negative entropy of the unannotated data $U=\langle u_{1}..u_{n}\rangle$ as shown below. A Gaussian prior is also added to the function.

Entropy Gradient Computation

Jiao et al. 2006 perform the computation of entropy gradient in the following manner:

As shown by them, the second term of the covariance is easy to compute. However, the first term requires calculation of quadratic feature expectations. The algorithm they proposed to compute this term involves $O(n^{2}s^{3})$ time complexity as it requires an extra nested loop in the forward-backward algorithm.

An alternative derivation proposed by Mann and McCallum, 2007 is presented as follows:

The second term in the summation is shown to cancel out in the above equation because,

$\sum _{Y}p_{\theta }(Y|x)\sum _{Y^{\prime }}p_{\theta }(Y^{\prime }|X)F_{k}(x,Y^{\prime })=\sum _{Y^{\prime }}p_{\theta }(Y^{\prime }|X)F_{k}(x,Y^{\prime })$

Hence, the gradient reduces to

The second term in the equation above can again be computed efficiently given the feature expectations obtained by forward/backward and the entropy for the sequence. The first term is now shown by the authors to be efficiently computable by performing further analysis. The first term is initially reduced to the following form:

In order to calculate this term, it is sufficient to compute $\sum _{Y_{-(i..i+1)}}p_{\theta }(Y|x)\log p_{\theta }(Y|x)$ for all pairs $y_{i}$ , $y_{i+1}$. This computation is performed by the then-introduced subsequence constrained entropy method.

Subsequence Constrained Entropy

Subsequence constrained entropy is defined as follows:

$H_{\sigma }\left(Y_{-(a..b)}|y_{a..b},x\right)=\sum _{Y_{-(a..b)}}p_{\theta }(Y|x)\log p_{\theta }(Y|x)$

The authors point that the entropy can be factored given a linear-chain CRF of Markov order 1, since $Y_{i+2}$ is independent of $Y_{i}$ given $Y_{i+1}$.

Computing the $H^{\alpha }(\cdot )$ and $H^{\beta }(\cdot )$ lattices is performed using the probabilities obtained by forward-backward in the following way:

The computational complexity of this calculation for one label sequence and subsequent iteration over all label pairs requires one run of forward-backward at $O(ns^{2})$, and equivalent time to calculate the lattices $H^{\alpha }(\cdot )$ and $H^{\beta }(\cdot )$ .