Alternating Minimization

From Cohen Courses
Jump to: navigation, search



Information Geometry and Alternating Minimization Procedures, by I. Csiszar, G.Tusnady. In Statistics and Decisions, 1984.


Alternating minimization (AM) is used when we are optimizing two variables jointly. Let us call these two variables p and q. The basic premise behind AM is that we keep one variable constant, say q and optimize the other variable, i.e., p, and then we keep p constant and optimize over q. This constitutes one cycle of the optimization procedure. One may find that AM is very similar to the EM algorithm, and in fact it has been shown by the original authors that EM is a special case of AM.

In recent work, this Method was presented in Soft-Supervised Text Classification as the primary method of solving or optimizing the objective function.


The alternating minimization algorithm attempts to solve a minimization problem of the following form:

given \mathcal{P}, \mathcal{Q} and a function D: \mathcal{P} \times \mathcal{Q}, we would like to find: \min_{(P, Q) \in \mathcal{P} \times \mathcal{Q}} D(P, Q) .

The way AM does this is by generating a sequence of variables (p^n, q^n) as follows: p^n = \arg\min_{\textrm{P}} D(p, q^{n-1}) , and q^n = \arg\min_{q} D(p^n, q) .

In order for AM to converge to the correct solution, certain conditions must be satisfied. These conditions can be encapsulated in a property known as the 5-points property. A theorem in the original paper states the conditions for convergence:

Theorem (Convergence of AM)

If p^{(n)} = \arg\min_{p \in \Delta^m} D(p, q^{n-1}), q^n = \arg\min_{q \in \Delta^m} D(p^n, q) and q_i^{(0)} (y) > 0 \quad \forall y \in Y, \forall i, then

  •  D(p,q) + D(p,p^{(0)}) \geq D(p, q^{(1)}) + D(p^{(1)}, q^{(1)}) \quad \forall p, q \in \Delta^m , and
  •  \lim_{n \to \infty} D(p^{(n)}, q^{(n)}) = \inf_{p, q \in \Delta^m} D(p,q)

where \Delta^m is the set consisting of m-length vectors, where each component in each vector lies within a |Y|-dimensional probability simplex, which is denoted as \Delta. The first point is the 5-point property, and the second point is the convergence statement.


As an example, let us look at what Subramanya and Bilmes did in their Soft-Supervised Text Classification paper. For them, (p^n, q^n) is a sequence of sets of distributions, i.e., every p^i is a set of distributions over all vertices in the graph. We know that their original formulation for their algorithm was:

 \min_{\textrm{p}} C_1(\textrm{p}) , where C_1(p) = \left[ \sum_{i=1}^l D_{KL} (r_i \parallel p_i) + \mu \sum_i^n \sum_j w_{ij}D_{KL}(p_i \parallel p_j) - \nu \sum_{i=1}^n H(p_i) \right].

In order to use alternating minimization, we introduce a second distribution over unlabeled data (apart from p), q. The new objective reads:

 \min_{\textrm{p, q}} C_2(\textrm{p, q}) , where C_2(p, q) = \left[ \sum_{i=1}^l D_{KL} (r_i \parallel q_i) + \mu \sum_i^n \sum_{j\in \mathcal{N}(i)} w'_{ij}D_{KL}(p_i \parallel q_j) - \nu \sum_{i=1}^n H(p_i) \right], where we define a new similarity matrix W' = W + \alpha \mathbf{I}_n, with  \alpha \geq 0 . We thus have two distributions defined for every training point, and the optimization procedure encourages the two to approach each other. Note that the smoothness regularizer (the third term, consisting of entropy) is defined on a different set of distributions as the first term, which aims to respect the labeled training data. It is this choice which makes the update equations for p^{(n)} and q^{(n)} relatively simple.

Assuming we start with a distribution that is initialized properly, we can express the update equations as:

 p_i^{(n)} (y) = \frac{1}{\mathcal{Z}_i} \exp\left\{ \frac{\beta_i^{(n-1)}(y)}{\gamma_i}\right\}, and  q_i^{(n)} (y) = \frac{r_i(y) \delta(i \leq l) + \mu \sum_j w'_{ji} p_j^{(n)} (y)}{\delta(i \leq l) + \mu \sum_j w'_{ji}} ,

where  \gamma_i = \nu + \mu \sum_j w'_{ij} and  \beta_i^{(n-1)} (y) = -\nu + \mu \sum_j w'_{ij} (\log q_j^{(n-1)} (y) -1) .

The derivations for these updates can be obtained by using the definitions of p^n and q^n provided above.

Related Papers