Wikipedia Infobox Generator Using Cross Lingual Unstructured Text
Team Members
Project Idea
Key facts about Wikipedia articles are often mentioned in short block sections called 'InfoBoxes'. For example, an article on Brazil would mention "Capital:Brasília", "Largest city:São Paulo", "Official language : Portuguese", etc. As these infoboxes are manually created & maintained, several articles have either missing or outdated information (that only got revised in the plain text). We also note that articles in languages closer to the native speakers have much more detailed information. For eg, articles on latin Soccer players have better facts like "Debut match, number of goals, Records wins" mentioned in Español version of Wikipedia whereas the English version lacks these crucial stats.
In our SPLODD project, we propose to achieve two objectives:
- Extract facts from unstructured wikipedia text to generate infobox
- Combine facts in multiple languages for an article to generate infoboxes with comprehensive information.
Methodology used:
- Topic modelling to detect type of page (Eg, country, person, athlete, actor)
- Training CRF models for fact identification and extraction
- Clustering for aligning multiple language entity names based on page topic
Challenges:
- Large scale data
- Noise : Same infobox entity name can be written in different ways in the same language. For eg, "Number of Goals","No. of Goals","Goals", "# Goals", etc.
Corpus
- Wikipedia XML Dumps (Current Revision only)
- http://en.wikipedia.org/wiki/Wikipedia_database#Other_languages
- English corpus size - 31 GB Uncompressed
- With 5-6 languages, approximately 200 GB total
Reference Papers
- Wu, Weld, Autonomously Semantifying Wikipedia, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management CIKM 2007
- Eytan Adar , Michael Skinner , Daniel S. Weld, Information arbitrage across multi-lingual Wikipedia, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
Comments from William
There's lots of cool space for a project here, and you guys are aware of the closely-related Weld work, which seems good. My only worry is the size of the data you're working with - do you have plans to boil this down to a size that's manageable for a class project? BTW might also want to look at the DBPedia dumps, which have regularized some of the infobox fields (using rules, I think), and distribute them. I don't know if there are non-English DBPedia dumps available.
--Wcohen 20:45, 22 September 2011 (UTC)
