View source for Davidov et al COLING 10

From Cohen Courses

← Davidov et al COLING 10

You do not have permission to edit this page, for the following reason:

---

You can view and copy the source of this page.

This a [[Category::Paper]] reviewed for Social Media Analysis 10-802 in Fall 2012. == Citation == author = {Dmitry Davidov and Oren Tsur and Ari Rappoport}, title = {Enhanced Sentiment Learning Using Twitter Hashtags and Smileys}, booktitle = {COLING (Posters)}, year = {2010}, pages = {241-249}, ee = {http://aclweb.org/anthology-new/C/C10/C10-2028.pdf}, crossref = {DBLP:conf/coling/2010p}, bibsource = {DBLP, http://dblp.uni-trier.de} == Online Version == [http://dl.acm.org/citation.cfm?id=1944566.1944594 Enhanced sentiment learning using Twitter hashtags and smileys] == Summary == The paper proposes a supervised framework for sentiment classification utilizing the Twitter dataset. The paper classifies the sentiment beyond the positive and negative labels by utilizing the 50 Twitter tags and 15 smileys as sentiment labels. The short textual sentences ( tweet) are sometimes labeled as sentiment tags, which assigns sentiment values to the tweet. The paper utilizes such tagged Twitter data for classification of a wide variety of sentiment types from text.our different kinds of features used and show that our framework successfully identifies sentiment types of the untagged tweets.relation between different emotions. F Automated identification of diverse sentiment types == Methodoloy == Features The features employed in classifying the sentiments can be broadly divided into 4 distinct types. - Single-word features They are considered as binary features with weight equal to the inverted count in the corpus - n-gram features The 2-5 length of consecutive words are considered as binary features with the same weighting as for the single-word features - Pattern-based features<math>Insert formula here</math> The words are classified as High frequency words (HFW) and content words (CW). A pattern is defined as an ordered sequence of HFW and slots for CW based on the frequency threshold. A pattern is defined as containing 2-6 HFWs and 1-5 slots for CWs. The weight for the pattern is assigned to the degree of match of the pattern - Exact match, Sparse match, Incomplete match and No match. - Punctuation features (1) Length of a sentence. Considering per sentence, (2) Number of exclamations, (3) Number of question marks, (4) Number of quotes, (5) Number of capital words. The weight assigned is the average weight of the respective features. Classification The algorithm used to assign the sentiment label to test examples is a slight modification of the k-NN algorithm. == Dataset == The dataset consists of 475 million public tweets from May 2009 to Jan 2010. All non-English characters are removed, and url links, hashtags and references have been replaced by URL/REF/TAG words. The content hashtags are treated as labels for the classification task. The sentiment labels are either hashtag-based or smiley-based. Hash-tag based labels - The frequent tags over the entire dataset were calculated and two human judges labeled them into five different categories. 1. strong sentiment, 2. most likely sentiment, 3. context-dependent sentiment, 4. focused sentiment and 5. no sentiment. The following table shows the annotation result. Smiley based labels - Amazon Mechnanical Turk (AMT) is used to obtian the list of commonly and unambiguous ASCII smileys. 50 hash-tag based of category strong sentiment and most likely sentiment along with 15 smiley based labels are considered as labels for the classification task. == Evaluation == *Evaluation using cross-validation The sentiment classification is evaluated using 10-fold cross-validation over the training set. The performance of the algorithm was tested under different feature settings. ** Multi-class classification There are 51 hashtag-based and 16 smiley based labels. The evaluation metric is the average f-score for 10-fold cross validation. The f-score for the random baseline is 0.02. The result is shown in the following table. The result is significantly better than the random baseline. ** Binary classification The labels are 1 if the sentence contains a particular label or 0 if the sentence does not bear any sentiment. For each of the 50 hashtag-based and 15 smiley-based labels, the binary classification is performed. The result is as shown in the following table. The results show that binary classification is better than the multi-class classification with a high precision value. * Evaluation with human judges Amazon Mechanical Turk (AMT) services was used to evaluate the performance of the classifier on test data. Te evaluation was considered correct if one of the tags selected by a human judge for a sentence was one of the 5 tags predicted by the algorithm. Th ecorrelation score for this task was \kappa = 0.41. == Observations == * This work presents a supervised classification framework for which utilizes Twitter hashtags and smileys as proxies for different sentiment types as labels. It contributes to avoiding the need for labor intensive manual annotation, allowing identification and classification of diverse sentiment types of short texts. * Binary classification of sentiments yields better results than multi-class classification. * Punctuation, word and pattern features contributes more towards classification performance, as compared to a small marginal boost provided by the the n-gram features. Pattern features provides better performance as compared to the combined effect of the rest of the features. * Explored inter-sentiment overlap and dependency by two simple techniques of tag occurrence and feature overlap. == Study Plan == K-Nearest Neighbours Multi-class classification == Related Work == McDonal et al (2007) Davidov and Rappoport (2006) Davidov and Rappoport (2008)

Return to Davidov et al COLING 10.