Reisinger et al 2010: Spherical Topic Models

Citation

Joseph Reisinger, Austin Waters, Bryan Silverthorn, and Raymond J. Mooney, "Spherical Topic Models", in Proceedings of the 27th International Conference on Machine Learning (ICML 2010), 2010.

Online version

Reisinger et al 2010

Summary

This is a recent paper that presents Spherical Mixture Model and Variational Inference methods for Latent Dirichlet Allocation (LDA), which is a Bayesian generative model for general problems in Topic modeling. The highlight of this paper is that it models documents as data points in high-dimensional spherical manifold. Like cosine similarity, the model assumes the data is directional, and can be parameterized by cosine distance and other similarity measures in directional statistics. The authors claim that the spherical topic modeling approach outperforms existing models such as LDA.

Motivations

Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA), fail to model the presence and the absence of words in the target document, because they assume multinomial distribution for document likelihood. To overcome this issue, the authors propose the Spherical Admixture Model, which models both the frequency as well as the presence and absence of the words. In addition to this, by assuming von Mises-Fisher distribution, they hope to improve the system accuracy when using high-dimensional spherical modeling methods for sparse text data.

Brief Description of the method

This paper first introduces the advantages of von Mises-Fisher distribution for text, then discusses the Spherical Admixture Model and the use of Variational Inference to solve the posterior approximation problem. In this section, we will first summarize the basic characteristics of von Mises-Fisher distribution they assume, then we will introduce the definition of the proposed model, as well as the variational inference method.

von Mises-Fisher Distribution

In LDA, the multinomial distribution of words assigns probabilities to integer vectors of event counts, which is the raw counts of each words in a document in $\mathbb {N} ^{|V|}$. In contrast to multimonial distribution, von Mises-Fisher (vMF) distribution is a probability distribution on the (d-1)-dimensional sphere in $\mathbb {R} ^{d}$, where its density function is

$f(\mathrm {X} ;\mu ,\kappa )=c_{d}(\kappa )exp(\kappa \mu ^{T}\mathrm {X} )$

where $\mu$ is the mean direction with $||\mu ||=1$, and $\kappa$ is the concentration parameter. In addition,

$c_{d}(\kappa )={\frac {\kappa ^{d/2-1}}{(2\pi )^{d/2}I_{{d/2}-1(\kappa )}}}$

is the normalization factor, where $I_{r}(\cdot )$ is the modified Bessel function of the first kind and order $r$.

Intuitively, vMF distribution can be considered as the multivariate Gaussian with spherical covariance, parametermized by the cosine distance rather than Euclidean distance. Cosine distance is commonly used in directional statistics and computes the directions of $l'_{2}$-normalized features vectors and corresponds to the normalized correlation coefficient.

In this paper, the authors also argue that vMF is sensitive to the absence/presence of words, where multinomial distribution is not. They showed an example: if document D1 has a vector of [1,1,1] and document D2 has a vector of [3,0,0], in multinomial scenario where topic proportion $\theta =[1/3,1/3,1/3]$, the two documents are equivalent. In contrast, vMF would compute different cosine distances.

The Spherical Admixture Model

The Spherical Admixture Model(SAM) is very different from LDA in the sense that it does not model each word given a topic distribution $P(w|z)$. Instead, it models the document, and uses a weighted directional average to combine topics. A simple generative story of SAM can be given by:

• Draw a set of T topics $\phi$ on the unit hypersphere.
• For each document d, draw topic weights $\theta _{d}$ from Dirichlet $\alpha$.
• Draw a document vector $v_{d}$ from vMF with mean ${\dot {\phi _{d}}}=Avg(\phi ,\theta _{d})$

The complete model can be represented as the following:

• $\mu |\kappa _{0}\sim vMF(m,\kappa _{0})$ (corpus mean)
• $\phi _{t}|\mu ,\epsilon \sim vMF(\mu ,\epsilon ),t\in T$ (topics)
• $\theta _{d}|\alpha \sim Dirichlet(\alpha ),d\in D$ (topic proportions)
• ${\dot {\phi _{d}}}|\phi ,\theta _{d}=Avg(\phi ,\theta _{d}),d\in D$ (spherical average)
• $v_{d}|{\dot {\phi _{d}}},\kappa \sim vMF({\dot {\phi _{d}}},\kappa )$ (documents)

Dataset and Experiment Settings

[[File:]]

Experimental Results

The authors performed three major experiments. The first experiment is the . The second experiment explores.

Exp

[[File:]]

Exp

[[File:]]

Exp

[[File:]]

Related Papers

This paper is related to many papers in three dimensions.

(1) .

(2) .

(3) .