Github Repo Recommendation:Topic Model meets Code
Contents
Comments
Looks like a nice, well-defined project. It's not immediately obvious to me how you plan to do the content-based recommendation. Are you going to use Blei's CTR? or is there some other approach you want to use?
Suggestions: the GraphLab matrix factorization code has been pretty widely used, and is fairly scalable
Good luck! --Wcohen 15:20, 10 October 2012 (UTC)
Task
Item recommendation
Overview
Github is a social network site for programmers, where they can host source code repositories (also called repos). Users can watch repos they are interested in; when a user is watching a repo, s/he will receive status updates on its activities (such as commits, tagging, etc…).
In 2009, Github hosted a recommendation contest, where the objective was to recommend repositories to users. The dataset contained 56K users, 120K repositories, and 440K user-watches-repo relationships between them.
Collaborative filtering typically uses some form of matrix factorization technique, and ignores content (in this case, the repos/code). Alternatively, in this project, we propose to incorporate inherent topics of the source code (of repos) to improve predictions.
Team
Naoki Orii and I'm looking for teammates
Datasets
- The dataset from the Github contest is available at https://github.s3.amazonaws.com/data/download.zip
Baseline Method
A collaborative-filtering based method, such as SVD
Challenges
- As it may be expensive to perform topic modeling on 120K repos, we may perform prediction on a subset of this data
More info about the Github contest
More information about the Github contest is available below:
- https://github.com/blog/466-the-2009-github-contest
- https://github.com/blog/481-about-the-github-contest
Related Work
Using topic models to improve collaborative filtering has been investigated in the following paper:
- C Wang and D Blei, "Collaborative Topic Modeling for Recommending Scientific Articles", in KDD 2011
The following papers describe applying topic models to code:
- E Linstead, P Rigor, S Bajracharya, C Lopes, and P Baldi, "Mining concepts from code with probabilistic topic models", ASE '07 Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering
- K Tian, M Revelle, and D Poshyvanyk, "Using latent dirichlet allocation for automatic categorization of software", MSR '09 Proceedings of the 2009 6th IEEE International Working Conference on Mining Software Repositories
