Kessler et al. 2009

Citation

Jason S. Kessler and Nicolas Nicolov. 2009. Targeting Sentiment Expressions through Supervised Ranking of Linguistic Configurations. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI '09).

Online version

Indiana CS

Summary

This paper examines four different methods of attaching sentimental-containing phrases (sentiment expressions) with the nouns they describe (mentions). They assume a separate module is able to accurately identify both sentiment expressions and mentions, and seek only to determine which sentiment expressions modify which mentions.

Furthermore, their algorithms build a graph-like structure linking the speaker or holder of the opinion with the sentiment expression, the sentiment expression with the mention, and meronyms of mentions with their parents. This allows for efficient extraction of results beyond just the single sentiment value of one mention. Instead, queries such as "What do people not like about product X?" or "What other features do users who dislike the camera’s zoom lens feel strongly about?" may be answered by their data representation.

Their data set consisted was constructed by them, and while their resulting precisions were lower than some other papers, their blog-based data set poses its own difficulties. Their methods did, however result in higher precision than other papers's methods on Kessler et Al.'s custom dataset.

Brief description of the method

This paper explores four methods for linking mentions and sentiment expressions.

The first is a simple baseline. We assume sentiment expressions and mentions are both already tagged. From there, we simply associate each sentiment expression with the mention closest to it. Should there be a tie, the one to the right is chosen.

The second system is called the Heuristic Syntax system. They leverage a dependency parse tree wherein they collapse all mentions and sentiment expressions to one node in the parse. Then, if a sentiment expression governs or is governed by a mention, they are associated.

The third system is based on work by Bloom, Garg, and Argamon, and is simply referred to as 'Bloom'. Again, this method leverages a dependency parse, as well as 42 hand-crafted rules produced by the aforementioned three authors. Each rule is a sequence of relational arcs in the dependency parse that is designed to lead from a sentiment expression to a mention. The rules are ordered such that the first rule of the list to match is supposed to be the best one.

Finally, the new work done by this paper is an approach called the RankSVM. The idea is to learn a model that for each sentiment expression will rank mentions in the same sentence according to their likelihood of being the target of the expression.

The authors use the SVMLight implementation with default parameters to create their SVM. They repeatedly point out that one large benefit to their system is that it can assign the same score to multiple instances, thus allowing one sentiment expression to connect to multiple mentions. This is particularly useful in the presence of coordinating conjunctions.

Experimental Result

On their own data set they showed an overall F-score of 69.8% with their best model, the RankSVM. The other methods scored from 44.5% with the Bloom method, up to 58.9% with the proximity-based model. The heuristic syntax model scored in between at 57.4%.

Interestingly, their baseline proximity-based model actually out performed both the Bloom and Heuristic Syntax. Their RankSVM model had very high precision, with 74.8%, but relatively low recall at just 65.4%.