Adamic, Zhang, Bakshy, and Ackerman, Knowledge Sharing and Yahoo Answers: Everyone Knows Something, WWW 2008

The Paper

ACM Portal Listing

Citation

L.A. Adamic, J. Zhang, E. Bakshy, and M.S. Ackerman, “Knowledge sharing and yahoo answers: everyone knows something,” Proceeding of the 17th international conference on World Wide Web, Beijing, China: ACM, 2008, pp. 665-674.

@inproceedings{www2010Adamic,
 author = {Adamic, Lada A and Zhang, Jun and Bakshy, Eytan and Ackerman, Mark S.},
 title = {Knowledge sharing and yahoo answers: everyone knows something},
 booktitle = {WWW '08: Proceeding of the 17th international conference on World Wide Web},
 year = {2008},
 isbn = {978-1-60558-085-2},
 pages = {665--674},
 location = {Beijing, China},
 doi = {http://doi.acm.org/10.1145/1367497.1367587},
 publisher = {ACM},
 address = {New York, NY, USA},
 }

Abstract from the paper

Yahoo Answers (YA) is a large and diverse question-answer forum, acting not only as a medium for sharing technical knowledge, but as a place where one can seek advice, gather opinions, and satisfy one's curiosity about a countless number of things. In this paper, we seek to understand YA's knowledge sharing activity. We analyze the forum categories and cluster them according to content characteristics and patterns of interaction among the users. While interactions in some categories resemble expertise sharing forums, others incorporate discussion, everyday advice, and support. With such a diversity of categories in which one can participate, we find that some users focus narrowly on specific topics, while others participate across categories. This not only allows us to map related categories, but to characterize the entropy of the users' interests. We find that lower entropy correlates with receiving higher answer ratings, but only for categories where factual expertise is primarily sought after. We combine both user attributes and answer characteristics to predict, within a given category, whether a particular answer will be chosen as the best answer by the asker.

Summary

Synopsis

In this paper, Adamic et al. conduct an analysis of Yahoo Answers (YA) using a mix of qualitative and quantitative methods. They tie their work back to a healthy literature of past work, then attempt to provide both a qualitative examination of some salient features of YA and a more detailed analysis of all the activity on the site during one month. From this, they develop a small grab bag of interesting features such as a k-means category clustering with some analysis of the different categories, a user entropy metric, and a model for predicting best answers.

Related Work

Adamic et al. make a point of specifically connecting their work to four different literatures of online interaction study:

1. Forum-focused
   • S. Whittaker, L. Terveen, W. Hill, and L. Cherny. The dynamics of mass interaction. Proceedings of the 1998 ACM conference on Computer supported cooperative work, pages 257-264, 1998.
   • K. Zhongbao and Z. Changshui. Reply networks on a bulletin board system. Phys. Rev. E, 67(3):036117, Mar 2003.
2. User-focused
   • E. Wegner. Communities of Practice: Learning, Meaning, and Identity, 1998.
   • J. Preece, B. Nonnecke, and D. Andrews. The top five reasons for lurking: improving community experiences for everyone. Computers in Human Behavior, 20(2):201-223, 2004.
3. Thread/Dialogue-focused
   • W. Sack. Conversation map: a content-based Usenet newsgroup browser. In IUI'00, pages 233-240, 2000.
   • E. Joyce and R. Kraut. Predicting Continued Participation in Newsgroups. Journal of Computer-Mediated Communication, 11(3):723-747 2006
4. Motivation-focused
   • K. Lakhani and E. von Hippel. How open source software works:\free" user-to-user assistance. Research Policy, 32(6):923-943, 2003.
   • B. Butler. Membership Size, Communication Activity, and Sustainability: A Resource-Based Model of Online Social Structures. Information Systems Research, 12(4):346-362, 2001.

Qualitative Notes on YA

The first points that Adamic et al. make about YA are that it is the most popular Q&A site in the US, and that it has 25 top level categories for questions with a growing list of subcategories beneath them. As with many Q&A sites threads are flat - meaning that while people may have discourse, it is in a technically stilted fashion as compared to the nested functionality that existed for USENET. That said, conversationality definitely appeared to be a key feature of certain categories on the site. Additionally some categories, such as "Programming", demonstrated a clear bias towards factual, specific respones. This is noted contrast with categories such as "Marriage & Divorce", in which answers may be quite subjective.

Analysis of Quantitative YA Data

The data

For their quantitative analysis of YA, Adamic et. al obtained a scrape of one month of activity on YA. They describe this data set as consisting of

includ[ing] 8,452,337 answers to 1,178,983
questions, with 433,402 unique repliers and 495,414 unique
askers. Of those users, 211,372 both asked and replied.

Categories

The paper initially looks into the nature of categories, arguing that a user's behavior can be seen as a function of the category in which she is operating. They also identify a few key category characteristics, based on a simple plot of thread length vs. post length:

• Categories with "factual" responses get fewer, longer answers.
• There are a few clear categories, such as "Baby Names" in which users provide many, short replies to threads.
• There is a block of categories with medium-length responses and medium-length threads which they term "discussion categories": these are things like "Wrestling" or "Immigration" where answers may exist but the purpose of the question is not necessarily to resolve any issues.

Clustering

Subsequent to making these observations, Adamic et al. used K-means clustering on thread length, content length, and the cosine similarity of the asking and replying frequencies for users within each category to more specifically identify three super-categories:

• "Discussion Forums": these posess a high proportion of users who both pose and answer questions and tend to attract long thread lengths - this essentially overlaps with with discussion categories mentioned above.
• "Indeterminant answers": Categories where people seek and provide advice and common-sense expertise on questions with indeterminate answers. These categories are again primarily characterized by thread length, though presumably the other elements of the model play some role.
• "Factual Answers": As noted above, these are the categories in which users tend to expect factual answers. This category has a lower cosine similarity than the others, as well as generally shorter threads with longer answers

The K-Means Categories: Red is informational, blue is indeterminant, and green is discussion-based.

Network Analysis

Adamic et al. next tried to approximate in- and out- link relationships for the different category clusters by examening interactions between users within the Wresting, Marriage & Divorce, and Programming categories - each subject proxied for its respective Category group (Discussion Forums, Indeterminant Answers, and Factual Answers). This primarily demonstrated that the three groups shuffle nto the logical order implied by their classification: Programming is low on both in- and out-degree, Wrestling is high, and Marriage & Divorce is in-between, nicely correlating with the factual-to-abstract nature of questions and also the perceived sociability of the categories.

• 

  Note that the distribution of in-links and out-links corresponds to expected ideas of group sociability

• 

  Note that information on the connected component matches the adjacent graph information.

Expertise sharing?

• In categories
• Across Categories - relationships between
• Mentions of yahoo products
• Asking connietion

User Entropy Metric

The authors point out that users do appear to cluster their activities among certain areas, and that different topics may feed into each other. As such, they argue that hey should look at user activity decompostion across different topics. To that end, the authors created an entropy metric, summing the different entropy values captured at the different tiers of the Yahoo category hierarchy: $\operatorname {H_{L}} =-\Sigma _{i}p_{L,i}log(p_{L,i})\operatorname {H_{T}} =\Sigma _{L}H_{L}$ where $H_{T}$ is the total entropy

This measure proves to be moderately useful for predicting whether a not a user will get a best answer for a question that need factual answers, still a useful predictor but a less powerful one for questions that fall into the independent answers category, and is not at all useful for calculating responses in general, response-oriented categories.

Statistical breakdown of Entropy as a predictor for three specific subjects that fall into the three k-Means subject classes.

Predicting Best Answers

The final section of the paper focuses on trying to predict best answers. To do so, the authors actually find it better to disregard the entropy measure previously discussed. Instead, they craft a logistic regression model and test it on a random subset of responses, some of which are marked as best and some of which are not. While initially including twelve different variables, the model was eventually constrained to four that were shown to be statistically significant predictors: reply length, thread length, the number of best answers by the user, and the number of replies ever given by the user. That said, the reply length may be the most important of the three predictors; the authors report that it gets a 62% accuracy all by itself, and the others variables do not cause a large amount of improvement. (The highest success rate is 72.9%).