Analyzing and Predicting Youtube Comments Rating: WWW2010
Contents
Citation
Stefan Siersdorfer, Sergiu Chelaru, Wolfgang Nejd, Jose San Pedro, "How useful are your comments?: analyzing and predicting youtube comments and comment ratings", Proceedings of the 17th international conference on World Wide Web WWW2010, 2010
Online version
Summary
This paper aims at analyzing comments made on videos hosted on Youtube, and predicting the ratings that users give to these comments. The ratings are basically number of people liking (positive rating) or disliking (negative rating) the comments made by other users. The authors refer to comments that have positive rating as accepted comments and those having negative ratings as unaccepted comments. The motivation is basically finding the sentiment of a comment, with the conjecture being that comment with "positive" sentiment tends to have positive rating, whereas one with "negative" sentiment tends to have negative rating. The authors also perform few other experiments to see the correlation between variance of ratings with polarity (more polar a video, more polar are people's opinions about it) of the videos, and the dependency of ratings and sentiment values of comments on videos of different categories. Please see the Youtube comment analysis dataset page for information about dataset.
Sentiment Analysis of Rated Comments
The authors first analyzed the comments for their sentiments to prove their hypothesis that positively rated comments have positive sentiment and vice-versa. They first categorized the comments into three categories "5Neg" (comments that have a negative rating of 5 or higher), "0Dist" (comments that have not got any rating) and "5Pos" (comments that have a positive rating of 5 or higher). Then the terms in these comments were assigned a sentiment score using SentiWordNet. SentiWordNet has a score triplet in the form of (positivity-score,negativity-score,objectivity-score) for each word present in WordNet. The authors just considered the adjectives present in the comments to be tagged for their sentiment scores. Experiments showed that negatively rated comments had more negative sentiment terms, and positively rated comments had more positive sentiment terms. Authors further did an "analysis of variance" test to prove that the mean of sentiment scores for the three categories varied significantly across any two categories.
Predicting Rating for Comments
After the above analysis, the authors did an SVM based classification of the comments. The comment was considered as a vector of sentiment values of the terms present in the comment. The classification was binary with the classes being positive/accepted or negative/unaccepted. For this experiment, authors considered distinct thresholds for the minimum and maximum ratings (above/below +2/-2, +5/-5, +7/-7) for comments to be considered accepted or unaccepted. The authors also chose different amounts of randomly chosen accepted and unaccepted comments (T=1000,10000,50000,200000) for training. At least 1000 comments in each of the classes were kept for testing. Three experiments were conducted. First was classification with accepted comments marked as accepted, and unaccepted comments marked as unaccepted. Second was classification with accepted comments marked as unaccepted, and unaccepted comments marked as accepted; this was done to find the "bad" or erroneous comments. The third experiment was with comments with high rating (positive or negative) and the ones with no rating. The three scenarios are labeled AC_POS, AC_NEG and THRES-0 in the results below.
Results for Rating Prediction
Variance of Comments Rating as Indicator of Polarizing topics
The authors also analyzed the relation between variance of comments rating and the polarity of videos. 1413 tags from 50 videos were selected and average variance of comment ratings was calculated over all videos having a particular tag. The table below shows top-25 and bottom-25 tags according to the average variance. We can see that tags in top-25 videos tend to be related to more polarizing topics, and the ones in bottom-25 videos tend to be related to rather neutral topics.
Category Dependencies of Ratings
Authors conducted the classification experiments separately for comments in three different categories: Music, Entertainment, and News & Politics. The results of these experiments is as shown in the figure below. While classification did comparably well for Entertainment and Music categories, it didn't do that well for News & Politics category.
References
[1] D. Shen, Q. Yang, J.-T. Sun, and Z. Chen. Thread detection in dynamic text message streams. In Proc. of SIGIR ’06, pages 35–42, Seattle, Washington, 2006.
[2] K. Jrvelin and J. Keklinen. IR evaluation methods for retrieving highly relevant documents. In Proc. of SIGIR ’00, pages 41–48, Athens, Greece, 2000.
