Ling and He Joint Sentiment Topic Model for Sentiment Analysis

Citation

author = {Lin, Chenghua and He, Yulan},
title = {Joint sentiment/topic model for sentiment analysis},
booktitle = {Proceedings of the 18th ACM conference on Information and knowledge management},
series = {CIKM '09},
year = {2009},
isbn = {978-1-60558-512-3},
location = {Hong Kong, China},
pages = {375--384},
numpages = {10},
url = {http://doi.acm.org/10.1145/1645953.1646003},
doi = {10.1145/1645953.1646003},
acmid = {1646003},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {joint sentiment/topic model, latent dirichlet allocation, opinion mining, sentiment analysis}

Online Version

Joint Sentiment/Topic Model for Sentiment Analysis

Summary

This paper proposes a probabilistic modeling framework based on Latent Dirichlet Allocation (LDA) which detects sentiment and topics simultaneously from text. Most approaches to sentiment analysis require labeled corpora for training and inference. The proposed model, however, is fully unsupervised i.e. does not require any labeled data as such. The authors hint towards why a joint model might is being sought. Sentiment polarities, intuitively, are dependent on topics or domains. For instance, though the adjective ‘unpredictable’ in a phrase such as ‘unpredictable steering’ has negative orientation in an automobile review, it has a positive orientation in a phrase like ‘unpredictable plot’ in a movie review.

The Joint Sentiment Topic (JST) model is similar in flavor to LDA the only major difference being an addition of a sentiment layer between the document and the topic layer. Each document in the Joint Sentiment Topic (JST) model is associated with distinct topic-document distributions corresponding to each of the $S$ sentiment labels. Each of these distributions correspond to a sentiment label $l$ with the same number of topics. The distribution defined by the topic and the sentiment label is then used to draw the word.

 1. For each document $d$, choose a distribution $\pi _{d}$ ~ Dirichlet($\gamma$)
 2. For each sentiment label $l$ under document $d$, choose a distribution $\theta _{d,l}$ ∼ Dir($\alpha$).
 3. For each word $w_{i}$ in document $d$
   a. Choose a sentiment label $l_{i}$ ∼ $\pi _{d}$
   b. Choose a topic $z_{i}$ ~ $\theta _{d,l_{i}}$
   c. Choose a word $w_{i}$ from the distribution over words defined by the topic $z_{i}$, sentiment label $l_{i}$ and another parameter $\phi _{z_{i}}^{l_{i}}$ 

The hyperparameter $\alpha$ can be treated as our prior belief about the number of times topic j was associated with sentiment label l sampled from a document and $\beta$ can be viewed as the number of times words sampled from topic j are associated with sentiment label l before having observed any actual words. Similarly, $\gamma$ can be interpreted as the prior observation counts for the number of times sentiment label l sampled from document before any words from the corpus is observed. In JST, there are three sets of latent variables to be inferred: the joint sentiment/topic-document distribution $\theta _{d,l}$, the joint sentiment/topic-word distribution $\phi _{z_{i}}^{l_{i}}$, and the sentiment-document distribution $\pi _{d}$.

Inference

A Gibbs sampling algorithm is provided for estimating the posterior distribution of the latent variables given a document. The algorithm sequentially samples each variable of interest from the distribution over that variable given the current values of all other variables and the data. The sequential update is carried on for a fixed number of Gibbs Sampling iterations

Tying-JST model

One has to choose a topic-document distribution $\theta _{d}$ for every document under the JST model. The tying-JST model simplifies this a bit. In the tying-JST model, there is only one topic-document distribution $\theta$ which accounts for all the documents in the corpus.

Experiments

The authors used a corpus of preprocessed Pang Movie Reviews for evaluating the performance of the JST model. The authors note that the sentiment classification problem is somewhat more challenging than the traditional topic-based classification, since sentiment can be expressed in a more subtle manner while topics can be identified more easily according to the co-occurrence of keywords. One of the directions for improving the sentiment detection accuracy is to incorporate some prior information in the model. In this paper, this was tried using 4 different methods:

• Paradigm word list

The paradigm word list consists of a set of positive and negative words which can be simply treated as paradigms for defining the positive and negative semantic orientation

• Mutual information

Mutual information was used to select words that have strong association with positive or negative sentiment classes. The top 20 words within each sentiment class were selected based on their MI scores and incorporated as prior information for the models.

• Full Subjectivity Lexicon

Publicly available subjectivity word list with established polarities was explored

• Filtered Subjectivity Lexicon

The filtered subjectivity lexicon was obtained by removing from the full subjectivity lexicon the words occurred less than 50 times in the movie review dataset. The words whose polarity changed after stemming were also removed automatically.

The authors evaluate both the JST and the tying-JST model against other existing models. Detailed experimental results of both document sentiment classification and topic extraction are provided in the paper. The results are quite elaborate to be provided here.

Study Plan

1. Blei_et_al_Latent_Dirichlet_Allocation