Predicting web searcher satisfaction with existing community-based answers

From Cohen Courses
Jump to navigationJump to search

This a Paper reviewed for Social Media Analysis 10-802 in Fall 2012.

Citation

author    = {Qiaoling Liu and
              Eugene Agichtein and
              Gideon Dror and
              Evgeniy Gabrilovich and
              Yoelle Maarek and
              Dan Pelleg and
              Idan Szpektor},
 title     = {Predicting web searcher satisfaction with existing community-based
              answers},
 booktitle = {SIGIR},
 year      = {2011},
 pages     = {415-424},
 ee        = {http://doi.acm.org/10.1145/2009916.2009974},
 crossref  = {DBLP:conf/sigir/2011},
 bibsource = {DBLP, http://dblp.uni-trier.de}

Online Version

Predicting web searcher satisfaction with existing community-based answers

Summary

The paper proposes a solution to a novel problem of predicting and validating the usefulness of Community-based Question Answering (CQA) sites for an external web searcher rather than an asker belonging to a community. The work has looked at three major components in the pipeline of solving the query satisfactionof users. They are as follows -

1. query clarity task - Whether a query is unambiguous enough to be interpreted as a question.

2. query-question match task - Measures the similarity between a query and a question.

3. answer quality - Assessing the sanctification of the answer with respect to the question in CQA, and thus indirectly relates to the satisfaction of the query.

The paper approaches the problem by building a regression model. The evaluation is performed by using human labeled data collected using crowdsourcing.

Methodoloy

Features

The features used for building the regression model has been divided according to the subtasks as mentioned above.

  • Query clarity features (9 total)
    • # of characters in the query.
    • # of words in the query.
    • # of clicks following the query.
    • Overall click entropy of the query.
    • User click entropy of the quer.
    • Query clarity score.
    • WH-type of the query - what,why,when,where,which,how,is,are,do.
  • Query-question match features(23 total)
    • Match score between te query and question title/body/answers using similarity metrics.
    • Jaccard/Dice/Tanimoto coefficient between the query and the question title.
    • Ratio between the number of characters/words in to the query to that in the question structure.
    • # of clicks on the question following this query.
    • # of users who clicked the question following thi/any query.
  • Answer quality features (37 total)
    • # of characters/words in the answer.
    • # of unique words in the answer.
    • # of answers received by the asker in the past.

For a full list of features please refer to the paper.

Models

  • Direct Logistic Regression

All the features listed above are using Logistic regression as the regressor model.

  • Composite Logisic Regression

A separate model is trained for each of he three subtasks mentioned above. The features pertaining to the subtask are used to train three separate regression models. The models are then combined into one by using Logistic Regression which assigns weights on the individual model.

Model.jpg

Evaluation

The evaluation is performed on Click Dataset on Google search leading to Yahoo! Answers.Evaluation measures - root mean square error(RMSE) and Pearson correlation between prediction and human judgement of query answer satisfaction.

  • Direct vs. Composite Comparison

The following table compares the direct logistic regression approach with composite logistic regression.

Table3.png

  • Answer ranking for queries

The following table compares the answer ranking with the method to that of the Google's ranking method.

Table2.png

Observations

  • This work presents a novel task of predicting the satisfaction of a Web searcher using the answers discussed in Community-based Question Answer(CQA) sites.
  • The modularization of the task into 3 sub tasks is a main feature of the paper. I liked this approach of solving the problem. In this case, the task were query clarity, query-question match and answer satisfaction.
  • The results show that the composite approach performs better than the direct approach. This is due to the additional information provided by the human judgments in each of the subtasks. Also, the performance of the individual components can be improved individually.
  • The ranking of the CQA answers generated by the work in the paper outperforms the ranking generated by the Google web engine with respect to the ground truth.
  • I like the problem described in the paper. However, the approach could also incorporate state-of-the-art langugae modeling techniques as well as IR-techniques. It would be an interesting analysis to show the change in the performance due to them.
  • Second, since an exhaustive list of features are used, it would be interesting to asess the relative importance of the features towards web searcher satisfaction.


Study Plan

Related Work

A similar work on retrieving content from social communities Finding high-quality content in social media..

Retrieved from "http://curtis.ml.cmu.edu/w/courses/index.php?title=Predicting_web_searcher_satisfaction_with_existing_community-based_answers&oldid=15488"