Ling and He Joint Sentiment Topic Model for Sentiment Analysis

Citation

author = {Lin, Chenghua and He, Yulan},
title = {Joint sentiment/topic model for sentiment analysis},
booktitle = {Proceedings of the 18th ACM conference on Information and knowledge management},
series = {CIKM '09},
year = {2009},
isbn = {978-1-60558-512-3},
location = {Hong Kong, China},
pages = {375--384},
numpages = {10},
url = {http://doi.acm.org/10.1145/1645953.1646003},
doi = {10.1145/1645953.1646003},
acmid = {1646003},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {joint sentiment/topic model, latent dirichlet allocation, opinion mining, sentiment analysis}

Online Version

Joint Sentiment/Topic Model for Sentiment Analysis

Summary

This paper proposes a novel probabilistic modeling framework based on Latent Dirichlet Allocation (LDA) which detects sentiment and topics simultaneously from text. Unlike other machine learning approaches to sentiment classification which often require labeled corpora for classifier training, the proposed model is fully unsupervised.

LDA

LDA is a generative probabilistic model for collections of discrete data such as text corpora. It is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying (latent) set of topics, where each topic is characterized by a distribution over words. Each document $d$ is assumed to be generated using the following process:

 1. Choose the number of words $N_{d}$ in the document by drawing from a distribution Poisson($\xi$)
 2. Choose the topic probabilities $\theta _{d,n}$ from a Dirichlet($\alpha$) distribution
 3. For each of the N words $w_{d,n}$
   a. Choose a topic $z_{d,n}$ from a Multinomial({$\theta {d,n}$) distrbution
   b. Choose a word $w_{d,n}$ from p($w_{d,n}|z_{d,n},\beta$) which is a multinomial distribution conditioned on the topic $z_{d,n}$

The parameters $\alpha$ and $\beta$ are corpus level parameters, assumed to be sampled once in the process of generating a corpus. The variables $\theta _{d,n}$ are document-level variables, sampled once per document. Finally, the variables $z_{d,n}$ and $w_{d,n}$ are word-level variables and are sampled once for each word in each document.

Inference

The posterior distribution of the hidden variables given a document is intractable. Efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation are provided.

The basic idea is to make use of Jensen’s inequality to obtain an adjustable lower bound on the log likelihood. A family of lower bounds, indexed by a set of variational parameters, is considered and the variational parameters are chosen by an optimization procedure that attempts to find the tightest possible lower bound. It leads to the following iterative EM algorithm

 1. E step: For each document, find the optimizing values of the variational parameters
 2. M step: Maximize resulting lower bound on the log likelihood with respect to the model parameters $\alpha ,\beta$ 

Experiments

LDA is empirically evaluated in several problem domains -- document modeling, document classification, and collaborative filtering.

Study Plan

1. Mixture models

2. Probabilistic Latent Semantic Indexing

3. Variational Bayesian Methods

4. Variational Inference lecture pdf by Blei