Tackstrom and McDonald, ECIR 2011. Discovering fine-grained sentiment with latent variable structured prediction models

Citation

O. Tackstrom and R. McDonald. 2011. Discovering fine-grained sentiment with latent variable structured prediction models. In Proceedings of $33^{rd}$ ECIR-2011, pp 764–773, Dublin, Ireland.

Online Version

Discovering fine-grained sentiment with latent variable structured prediction models

Summary

This paper investigates the use of latent variable structured prediction models for fine-grained sentiment analysis in the common situation where only coarse-grained supervision is available. The authors show how sentence level sentiment labels can be effectively learned from document-level supervision using hidden conditional random fields (HCRFs). The authors show improvements over both lexicon and existing machine learning based approaches. They focus on sentence level sentiment analysis.

Method

The authors observe that there is a lot of data in the form of coarse-level annotations available on the web pertaining to consumer reviews of products, movies etc. However, fine-grained labeled data for sentiment is difficult to obtain across domains for supervised learning. Hence, the authors model finer-level information as latent variables making use of the freely available coarse level annotations, using hierarchical graphical models such as HCRFs.

Based on the observations about positive and negative reviews in documents, the authors model sentence level classifications as:

• Correlated with the observed document label and,
• Flexible enough to disagree when contextual evidence suggests otherwise.

Approach

They start with the supervised fine-to-coarse sentiment model described in McDonald et al., 2007.

Let $d$ be a document consisting of $n$ sentences, ${\textbf {s}}=(s_{i})_{i=1}^{n}$ Let the document level sentiment and sentence level sentiment be denoted by ${\textbf {y}}^{d}=(y^{d},{\textbf {y}}^{s})$ be the random variables that include the document level sentiment, $y^{d}$, and the sequence of sentence level sentiment, ${\textbf {y}}^{s}=(y_{i}^{s})_{i=1}^{n}$

All random variables take values in $\{POS,NEG,NEU\}$ for positive, negative and neutral sentiment, respectively. The authors hypothesize that there is a sequential relationship between sentence sentiment and that the document sentiment is influenced by all sentences (and vice versa). A first order Markov property is assumed, according to which each sentence variable, $y_{i}^{s}$ is independent of all other variables, conditioned on the document variable $y_{d}$ and its adjacent sentences, $y_{i-1}^{s},y_{i+1}^{s}$.

The graphical model for the following formulation is represented in the figure below:

In the figure above, a graphical model with latent sentence level states is shown. Dark grey nodes are observed variables and white nodes are unobserved. Light grey nodes are observed at training time. Dashed and dotted regions indicate the maximal cliques at position $i$.

In the HCRF model above, the conditional probability of the observed variables is obtained by marginalizing over the posited hidden variables, given as, $p_{\theta }(y^{d}|{\textbf {s}})=\sum _{{\textbf {y}}^{s}}p_{\theta }(y^{d},{\textbf {y}}^{s}|{\textbf {s}}).$

As indicated in the figure above, there are two maximal cliques at each position $i$. One involving only the sentence $s_{i}$ and its corresponding latent variable $y_{i}^{s}$ and one involving the consecutive latent variables $y_{i}^{s},y_{i-1}^{s}$ and the document variable $y_{d}$.

The assignment of the document variable $y_{d}$ is thus independent of the input ${\textbf {s}}$, conditioned on the sequence of latent sentence variables ${\textbf {y}}^{s}$. This distinction is important for learning predictive latent variables as it creates a bottleneck between the input sentences and the document label.

It was observed that while training HCRFs, using hard estimation gave slightly better performance as opposed to doing MAP estimate of the parameters with respect to the marginal conditional log-likelihood of observed variables, assuming a Normal prior distribution.

Assuming $D={s_{j},y_{j}^{d}}_{j=1}^{m}$ as the training set of document/document-label pairs, the parameter $\theta$ is estimated in the following manner for HCRFs:

In $(1)$, parameter $\theta$ can be estimated by using the stochastic gradient descent algorithm for 75 iterations with a fixed step size $\eta$.

Viterbi algorithm is used in equation $(2)$ in the predicting the optimal assignment of $(y^{d},{\textbf {y}}^{s})$.