GeneralizedIterativeScaling

This is one of the earliest methods used for inference in log-linear models. Though more sophisticated and faster methods have evolved, this method provides an insight in log linear models.

What problem does it address

The objective of this method is to find a probability function of the form

$(1)\quad \quad p_{i}=\pi _{i}\mu \prod _{s=1}^{d}\mu _{s}^{b_{si}}$

satisfying the constraints

$(2)\quad \quad \sum _{i\in I}b_{si}p_{i}=k_{s}$

where $I$ is an index set; the probability distribution over which has to be determined, $p$ is a probability distribution and $\pi$ is a subprobability function (adds to 1 but $\pi _{i}\neq 0$ for any $i$); $b_{si}\neq 0$ is constant.

Existence of a solution

If $p$ of form (1) exists satisfying (2), then it minimizes $KL[p,\pi ]$ and is unique. Since $\pi _{i}$ are constant; it essentially boils down to the following statement.

Maximizing Entropy

If there exists a positive probability function of the form

$p_{i}=\mu \prod _{s=1}^{d}\mu _{s}^{b_{si}}$

satisfying (2), then it maximizes the entropy

$H(p)=-\sum _{i}p_{i}log(p_{i})$