Semantic Role Labeling with CRFs
Contents
Citation
Trevor Cohn, Philip Blunsom, "Semantic Role Labeling with Conditional Random Fields", CoNLL 2005
Online version
Summary
This paper aims at Semantic Role Labeling of sentences using Conditional Random Fields. The random field is
Sentiment Analysis of Rated Comments
The authors first analyzed the comments for their sentiments to prove their hypothesis that positively rated comments have positive sentiment and vice-versa. They first categorized the comments into three categories "5Neg" (comments that have a negative rating of 5 or higher), "0Dist" (comments that have not got any rating) and "5Pos" (comments that have a positive rating of 5 or higher). Then the terms in these comments were assigned a sentiment score using SentiWordNet. SentiWordNet has a score triplet in the form of (positivity-score,negativity-score,objectivity-score) for each word present in WordNet. The authors just considered the adjectives present in the comments to be tagged for their sentiment scores. Experiments showed that negatively rated comments had more negative sentiment terms, and positively rated comments had more positive sentiment terms. Authors further did an "analysis of variance" test to prove that the mean of sentiment scores for the three categories varied significantly across any two categories.
Predicting Rating for Comments
After the above analysis, the authors did an SVM based classification of the comments. The comment was considered as a vector of sentiment values of the terms present in the comment. The classification was binary with the classes being positive/accepted or negative/unaccepted. For this experiment, authors considered distinct thresholds for the minimum and maximum ratings (above/below +2/-2, +5/-5, +7/-7) for comments to be considered accepted or unaccepted. The authors also chose different amounts of randomly chosen accepted and unaccepted comments (T=1000,10000,50000,200000) for training. At least 1000 comments in each of the classes were kept for testing. Three experiments were conducted. First was classification with accepted comments marked as accepted, and unaccepted comments marked as unaccepted. Second was classification with accepted comments marked as unaccepted, and unaccepted comments marked as accepted; this was done to find the "bad" or erroneous comments. The third experiment was with comments with high rating (positive or negative) and the ones with no rating. The three scenarios are labeled AC_POS, AC_NEG and THRES-0 in the results below.
Results for Rating Prediction
Variance of Comments Rating as Indicator of Polarizing topics
The authors also analyzed the relation between variance of comments rating and the polarity of videos. 1413 tags from 50 videos were selected and average variance of comment ratings was calculated over all videos having a particular tag. The table below shows top-25 and bottom-25 tags according to the average variance. We can see that tags in top-25 videos tend to be related to more polarizing topics, and the ones in bottom-25 videos tend to be related to rather neutral topics.
Category Dependencies of Ratings
Authors conducted the classification experiments separately for comments in three different categories: Music, Entertainment, and News & Politics. The results of these experiments are as shown in the figure below. While classification did comparably well for Entertainment and Music categories, it didn't do that well for News & Politics category. In addition, authors also found mean rating scores for comments in a variety of categories. The results are as in the figure below.
