Topic Model Approach to Authority Identification

This a Paper reviewed for Social Media Analysis 10-802 in Fall 2012.

Citation

 author    = {Alexandre Passos and 
              Jacques Wainer and
              Aria Haghighi},
 title     = {What do you know? A topic-model approach to authority identification},
 journal   = {NIPS 2010 Workshop on Computational Social Science and the Wisdom of the Crowds},
 year      = {2010}

Online version

What do you know? A topic-model approach to authority identification

Summary

In this paper the authors present a preliminary study of basic approaches to the problem of identifying authoritative documents in a given domain using textual content and report their best performing approach using Hierarchical Topic Models [Blei et al, 2004]. Authoritative documents are ones which exhibit novel and relevant information relative to a document collection while demonstrating domain knowledge. Authors define authoritativeness identification task as a ranking problem and focus on product (book GoodReads and restaurant Yelp) reviews utilizing user votes as proxy for helpfulness and authoritativeness.

Dataset Description

The authors have reported results on two datasets.

• Book Reviews GoodReads Dataset
  • First 326 books in the "Best Books Ever" Category
  • First 60 odd reviews from each book.
• Restaurant Reviews Yelp Dataset
  • 283 Most reviewed restaurants in the Boston/Cambridge area

Number of "helpful" user votes for each review were considered as a proxy for ranking reviews authoritativeness.

Task Description and Evaluation

Models

Heuristic Approaches

• random : Reviews sorted randomly
• nwords : Reviews sorted by number of votes [More votes more authoritative]
• unique : For each word w, let $g_{w}$ be its count across all documents for all products. Let $p_{w}$ be its count among documents of a given product. Review d of this product are then ranked by the number of words unique among the document collection. The score associated with a document is, $\Sigma _{w\in ds.t.p_{w}=1}log(g_{w}+1)$

Summarization-Based Approaches

• sumbasic: Rank documents by the sum-basic criterion [Nenkova and Vanderwende, 2005], ordering reviews of the same product by how many high-frequency words they have relative to the product document collection. The score of a document D is $\Sigma _{w\in dP(w)}$.
• klsum: Rank by the kl-sum criterion [Haghighi and Vanderwende 2009]. Ranking by the unigram KL divergence $KL(P_{p}||P_{r})$ , where $P_{p}$ is a smoothed distribution for all reviews of the same product, $P_{r}$ is a smoothed distribution for each review. Both distributions are drawn from a symmetric Dirichlet with hyper-parameter 0.01.

Discriminative Approach

• logreg: A regularized Logistic regression classifier, trained to pick the best review for each single product versus the bottom 30%. The authors used L2 regularization with $\sigma ^{2}=5$ and the L-BFGS optimizer [Byrd et al., 1995]

Topic Model

Variation of hierarchical LDA [Blei et al, 2004] with a fixed tree structure. Model assumes each word in each review either comes from a distribution common to all reviews or from a product specific content distribution common to all reviews of the same product. Ranking is then performed on the number of rare content words, $\Sigma _{w\in p}{\frac {1}{df_{w}}}$, where $df_{w}$ is number of reviews of the product that used word w.

Results

Metric Used:

• Precision@K = ${\frac {k}{c}}$ , c is the number of reviews one has to look at to find the K Best reviews.
• NDCG@K : Normalized discounted cumulative gain of the first K elements of the ranked list.
• Average Rank of the best review
• Normalized Average Rank of the best review

Findings

• Though the topic modeling approach performed the best when compared to other baseline methods, the authors report two major issues with the approach. First, often many irrelevant reviews (like spams) were very well ranked owing to the fact that they were introducing new but unusual words to the distribution leading to a higher score. Secondly, there intuition on mixing between a topic model to select words and tf-df heuristic to rank documents is unusual.
• The paper seems to be a preliminary work by the authors in this domain and it seems there was no follow up work indicating that the approach was not continued with
• I wont recommend the paper for future reading. It lacks details and fails to achieve a satisfactory conclusion at the end. As also reported by the authors, current model ranks reviews based on the number of rare content words which causes irrelevant or spam reviews to be ranked higher. One can use other sources of information like author identity, star ratings, etc to improve upon these results.

Related papers

• Understanding Hierarchical Topic Models.

D. Blei, T.L. Griffiths, M.I. Jordan, and J.B. Tenenbaum. 2004. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems, 16:106.

• Understanding the sum-basic criterion.

A. Nenkova and L. Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005-101.

• Understanding the KL-SUM criterion

A. Haghighi and L. Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of HLT-NAACL 2009, pages 362–370. Association for Computational Linguistics.

Study plan

• Hierarchical Topic Models
  • Chinese Restaurant Process
  • Gibbs Sampling
• KL Divergence
• Logistic Regression
  • L2 Regularization