Topics over Time

Summary

Topics over Time (TOT) is an LDA-style method for topic modeling that explicitly models word co-occurrences jointly with time. In other words, TOT captures both the low-dimensional structure of data and how the structure changes over time. This method is motivated by the realization that structure in data (in this case topics co-occurrence patterns) are not static but dynamic. Patterns present in the early part of the data may not be in effect later. Topics rise and fall, split and merge, and change correlations over time. In TOT, each generated document has a mixture model of topics that is influenced jointly by both word co-occurrences and the document's time stamp.

Procedure

For each generated document, mixture distribution over topics in the document is jointly influenced by word co-occurrences and the document's time stamp.

In LDA style of generative model, TOT first draws T multinomials $\phi _{z}\,\!$ from a Dirichlet prior $\beta \,\!$, one for each topic z.

Then, for each document d, TOT draws a multinomial $\theta _{d}\,\!$ (the distribution of topics specific to the document d) from a Dirichlet prior $\alpha \,\!$.

Then, for each word $w_{di}\,\!$ (the i th token in document d):

• TOT draws a topic $z_{di}\,\!$ (the topic associated with the i th token in document d) from the multinomial $\theta _{d}\,\!$
• TOT draws a word $w_{di}\,\!$ from the multinomial $\phi _{z_{di}}\,\!$ (the distribution of words specific to topic z)
• TOT draws a time stamp $t_{di}\,\!$ (the time stamp associated with the i th token in the document d) from Beta $\psi _{z_{di}}\,\!$ (the beta distribution of time, specific to topic z)

In this generative process, a time stamp is associated with each word, which is perfect when different parts of the document are discussing different time periods. However, common documents typically have only one time stamp per document. Therefore, an alternative model to TOT is one where a single time stamp is associated with each document (figure below):

Using this model, given the input: the words in the document, we can output: a prediction of the time stamp of the document. To facilitate this prediction, time stamps are first discretized. Then, given a document, the model predicts its time stamp by maximizing the posterior which is the product of time stamp probability of all word tokens from their corresponding topic-wise Beta distributions over time: $\textstyle argmax_{t}\prod _{i=1}^{N_{d}}p(t|\psi _{zi})\,\!$, where $N_{d}\,\!$ is the number of word tokens in the document.

Related Methods

The strength of TOT is in its joint modeling of time and word co-occurrence. Unlike TOT, rather than doing joint modeling, other methods that model changes of topics over time do it "non-jointly"; either by (1) first fitting a time-unaware topic model on data and then ordering the documents in time, or (2) divides data into discrete time slices and fits a separate topic model in each slice. The disadvantage of such approach is that they are not making full use of time information to improve topic discovery.

Unlike other topic modeling methods that treat the "meaning" (or word distributions) of topics as changing over time, topics in TOT are modeled as constant over time. It is the occurrence and co-occurrence of the topics that are changing over time, not the word distribution of the topics. The rise and fall of topics over time, the split and merge of topics over time, are modeled as changes in the occurrence and co-occurrence of the topics over time.

Other related method is "burst of activity model" in Peak Detection that uses probabilistic infinite-state automaton with a Markov model of state structure in which high activity states are reachable only by passing through lower activity states. This method is not working directly with time stamps, rather use data ordering as a proxy for time. Unlike this method, TOT uses time stamps directly. Also, instead of employing Markov assumption over time, TOT treats time as an observed continuous variable.

References / Links

• Wang, X. and McCallum, A. Topics over Time: A Non-Markov Continuous-Time Model of Topical Trends, KDD 2006. - [1]