Comparison: O'Connor et al. ICWSM 2010 & Widespread Worry and Stock Market

Papers

• OConnor et. al., ICWSM 2010 Brendan O’Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010a. From tweets to polls: Linking text sentiment to public opinion time series. In Proc. of ICWSM.
• Widespread Worry and the Stock Market Gilbert, E. and Karahalios, K., Widespread worry and the stock market, 2010, In Proceedings of the international conference on weblogs and social media (ICWSM 10).

Method

The two papers used different methodologies. The O'Conner et al. paper studied sentiment analysis of Twitter data by using a sentiment lexicon to mark positive or negative tweets, whereas the Gilbert et al. paper used supervised learning classifiers to automatically identify anxious LiveJournal blog posts. Furthermore, Gilbert et al. analyzed the usefulness of time series data from their Anxiety index in forecasting S&P 500 stock market data using Granger causality, whereas O'Connor et al. simply studied the smoothed correlation between public opinion polls and their aggregated Twitter sentiment.

Dataset Used

• Gilbert et al. used two data sets: LiveJournal blog dataset and the S&P 500 dataset.
• O'Connor et al. uses a dataset consisting of roughly 1 billion Twitter posts collected between 2008-2009.

Problem

Gilbert et al. was concerned with finding a blog-based economic indicator that could be used to predict movements of the S&P 500.

O'Connor et al. was concerned with using aggregated sentiment of Twitter posts as an alternative (or supplement) to traditional public opinion polling, for such areas as consumer confidence or presidential approval ratings.

Big Idea

The big idea common to both papers is that public sentiment can be roughly estimated from large natural language crowd-sourced textual data (LiveJournal blogs and Twitter). The high-level goal of both papers is to use the textual data to estimate the latent sentiment of the general population, and use these estimates as alternatives to costly social science polling.

Other

The correlations reported in the O'Connor et al. paper seemed fairly significant (> 75% in many cases), whereas the results in the Gilbert et al. paper were not as strong, with fairly large p-values in many cases.

Additional Questions

How much time did you spend reading the (new, non-wikified) paper you summarized? 1 hour

How much time did you spend reading the old wikified paper? 1 hour

How much time did you spend reading the summary of the old paper? 20 minutes

How much time did you spend reading background material? 20 minutes

Was there a study plan for the old paper? Yes

If so, did you read any of the items suggested by the study plan? and how much time did you spend with reading them? No.

Overall, I found the previous summary useful for getting a high-level understanding of what the O'Connor et al. paper was about, and it helped me know what to look for in terms of comparisons when reading the new paper. To get an understanding of the actual methodology and relevant facts concerning the old paper, however, required me to read it.