Birke&Sarkar,FigLanguages07

Citation

title = {Active learning for the identification of nonliteral language},
author = {Birke, Julia and Sarkar, Anoop},
booktitle = {Proceedings of the Workshop on Computational Approaches to Figurative Language},
series = {FigLanguages '07},
year = {2007},
location = {Rochester, New York},
pages = {21--28},

Abstract from the paper

In this paper we present an active learning approach used to create an annotated corpus of literal and nonliteral usages of verbs. The model uses nearly unsupervised word-sense disambiguation and clustering techniques. We report on experiments in which a human expert is asked to correct system predictions in different stages of learning: (i) after the last iteration when the clustering step has converged, or (ii) during each iteration of the clustering algorithm. The model obtains an f-score of 53.8% on a dataset in which literal/nonliteral usages of 25 verbs were annotated by human experts. In comparison, the same model augmented with active learning obtains 64.91%. We also measure the number of examples required when model confidence is used to select examples for human correction as compared to random selection. The results of this active learning system have been compiled into a freely available annotated corpus of literal/nonliteral usage of verbs in context.

Online version

pdf link to the paper

Summary of approach

• The main goal of this article is to automatically create a corpus of literal vs. nonliteral usages of verbs. For example, given the verb ‘pour’, the algorithm should annotate the sentence ‘Custom demands that cognac be poured from a freshly opened bottle’ as literal, and ‘Salsa and rap music pour out of the windows’ as nonliteral

• The problem of nonliteral language recognition is reduced to one of word-sense disambiguation (WSD) by redefining ‘literal’ and ‘nonliteral’ as two senses of the same word. Since there is no annotated data to train a classifier that uses local context around the verb as features of literal/nonliteral examples, authors use nearly unsupervised approach, augmented with active learning.

• In a first stage of the algorithm, authors use a weakly-supervised method to separate literal from non-literal usages, described in (Birke & Sarkar, 2006). The operation is supported by two seed sets that contain examples representing both situations: the literal feedback seed set contains data from the Wall Street Journal Wall Street Journal (WSJ) corpus and the non-literal feedback seed set is composed of idiomatic and metaphoric expressions taken from dictionaries. Then, for a given sentence containing a verb to be tested, a word-based comparison is performed against all the sentences of each feedback set. The sentence is classified as either literal or non-literal, taking into account the set in which the most similar sentence was found.

• In a second stage, the method is improved by using an active learning strategy, in which the learner has the ability to influence the selection of a part of a training data. In each iteration the clustering algorithm sends a small set of examples (algorithm uses at most 30% of examples that it could not classify due to low confidence in the labeling) to a human expert to annotate, and uses the annotated data in the next iteration.

Experiments and results

The model was evaluated on 25 target verbs:

absorb, assault, die, drag, drown, escape, examine, fill, fix, flow, grab, grasp, kick, knock, lend, miss, pass, rest, ride, roll, smooth, step, stick, strike, touch

Authors manually annotated 1 to 115 sentences per verb into two clusters: literal and nonliteral. The algorithm was evaluated based on how accurately it clustered the hand-annotated sentences.

• In the first experiment (clustering without active learning) the models obtains an f-score of 53.8%
• The model augmented with active learning obtains 64.9%

The result of this work is the TroFi (Trope Finder) Example Base [1] of literal and nonliteral usage for fifty verbs which occur in 3,737 sentences from the Wall Street Journal (WSJ) corpus. In each sentence, the target verb is labeled L (literal) or N (nonliteral), according to the sense of the verb that is invoked by the sentence.

Related Papers

• A Clustering Approach for the Nearly Unsupervised Recognition of Nonliteral Language. Julia Birke and Anoop Sarkar. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, EACL-2006. Trento, Italy. April 3-7, 2006. pdf

Study Plan

Papers you may want to read to understand this paper.

• Active Learning Literature Survey by Burr Settles pdf

• Word sense disambiguation: a survey (2009) by Roberto Navigli pdf