Difference between revisions of "Tomanek 2009 semi supervised active learning for sequence labeling"

From Cohen Courses
Jump to navigationJump to search
m (1 revision)
Line 1: Line 1:
{{MyCiteconference | booktitle = Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the ACL| coauthors = U. Hahn| date = 2009| first = K.| last = Tomanek| pages = 1039-1047| title = Semi-Supervised Active Learning for Sequence Labeling| url = http://www.seas.upenn.edu/~strctlrn/bib/PDF/cpi_conll2006_camera.pdf }}
+
== Citation ==
  
This [[Category::Paper]] is available online [http://www.seas.upenn.edu/~strctlrn/bib/PDF/cpi_conll2006_camera.pdf].
+
Tomanek, K. and Hahn, U. 2009. Semi-Supervised Active Learning for Sequence Labeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the ACL.  
  
== Reviews of this paper ==
+
== Online version ==
{{#ask: [[reviewed paper::tomanek_2009_semi_supervised_active_learning_for_sequence_labeling]] | ?reviewer}}
+
 
 +
[http://www.aclweb.org/anthology/P/P09/P09-1117.pdf Link to ACLWeb]
 +
 
 +
== Summary ==
 +
 
 +
This [[Category::paper]] presents an [[UsesMethod::Active Learning]] approach that is not fully supervised. In this paper, the authors propose a semi-supervised approach where only some of the sequences that are highly uncertain are not labeled and others are automatically labeled.
 +
 
 +
== Brief description of the method ==
 +
 
 +
The method is a pretty simple extension of a standard active learning method. The following figure describes the general active learning framework.
 +
 
 +
[[File:Tomanek ACL2009.png]]
 +
 
 +
The authors refer the usual active learning mode as fully supervised active learning (FuSAL). The utility function used in FuSAL is
 +
 
 +
<math>U_{\mathbf{\lambda}}(\mathbf{x}) = 1 - P_{\mathbf{\lambda}}(\mathbf{y}^{*}\vert\mathbf{x})</math>
 +
 
 +
which makes the sampling method as an uncertainty sampling method.
 +
 
 +
The problem of FuSAL in the sequence labeling scenario is that an example that has a high utility can still have parts of it that the current model can label very well, thus not contribute much to the utility of the whole. Therefore, it means we can leave some of the labels that the current model labeled if the confidence on that particular token is high enough. The authors name this as semi-supervised active learning (SeSAL). It combines the benefits of [[UsesMethod::Active Learning]] and [[UsesMethod::Bootstrapping]], which are labeling only examples with high utility and minimizing annotation effort by partially labeling examples where the model is confident about the prediction. In pseudocode, the following shows the steps that are added to the FuSAL:
 +
 
 +
3.1 For each example <math>p_{i}\quad</math>
 +
 
 +
3.1.1 For each token <math>x_{j}\quad</math> and the most likely label <math>y_{j}^{*}\quad</math>
 +
 
 +
3.1.1.1 Compute the model's confidence in the predicted label <math>C_{\mathbf{\lambda}}(y_{j}^{*})=P_{\mathbf{\lambda}}(y_{j}=y_{j}^{*}\vert\mathbf{x})</math>
 +
 
 +
3.1.2 Remove all labels whose confidence is lower than some threshold <math>t</math>
 +
 
 +
Since there is a bootstrapping element in the method, the size of the seed set is also important. Therefore the authors suggest running FuSAL several iterations before switching to SeSAL.
 +
 
 +
== Experimental Result ==
 +
 
 +
The authors tested this method on [[UsesDataset::MUC]]-7 and the oncology part of [[UsesDataset::PennBioIE]] corpus. The base learner used for the experiment is a linear-chain [[UsesMethod::Conditional Random Fields]]. Features used are orthographical features (regexp patterns), lexical and morphological features (prefix, suffix, lemmatized tokens), and contextual features (features of neighbor tokens). In terms of the number of tokens that had to be labled to reach the maximal F-score, SeSAL could save about 60% over FuSAL, and 80% over random sampling. Having high confidence was also important because it could save the model from making errors in the early stages.
 +
 
 +
== Related papers ==
 +
 
 +
* [[RelatedPaper::Muslea, Minton and Knoblock, ICML 2002]]
 +
* [[RelatedPaper::McCallum and Ngiam, ICML 98]]

Revision as of 22:32, 30 September 2011

Citation

Tomanek, K. and Hahn, U. 2009. Semi-Supervised Active Learning for Sequence Labeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the ACL.

Online version

Link to ACLWeb

Summary

This paper presents an Active Learning approach that is not fully supervised. In this paper, the authors propose a semi-supervised approach where only some of the sequences that are highly uncertain are not labeled and others are automatically labeled.

Brief description of the method

The method is a pretty simple extension of a standard active learning method. The following figure describes the general active learning framework.

Tomanek ACL2009.png

The authors refer the usual active learning mode as fully supervised active learning (FuSAL). The utility function used in FuSAL is

which makes the sampling method as an uncertainty sampling method.

The problem of FuSAL in the sequence labeling scenario is that an example that has a high utility can still have parts of it that the current model can label very well, thus not contribute much to the utility of the whole. Therefore, it means we can leave some of the labels that the current model labeled if the confidence on that particular token is high enough. The authors name this as semi-supervised active learning (SeSAL). It combines the benefits of Active Learning and Bootstrapping, which are labeling only examples with high utility and minimizing annotation effort by partially labeling examples where the model is confident about the prediction. In pseudocode, the following shows the steps that are added to the FuSAL:

3.1 For each example

3.1.1 For each token and the most likely label

3.1.1.1 Compute the model's confidence in the predicted label

3.1.2 Remove all labels whose confidence is lower than some threshold

Since there is a bootstrapping element in the method, the size of the seed set is also important. Therefore the authors suggest running FuSAL several iterations before switching to SeSAL.

Experimental Result

The authors tested this method on MUC-7 and the oncology part of PennBioIE corpus. The base learner used for the experiment is a linear-chain Conditional Random Fields. Features used are orthographical features (regexp patterns), lexical and morphological features (prefix, suffix, lemmatized tokens), and contextual features (features of neighbor tokens). In terms of the number of tokens that had to be labled to reach the maximal F-score, SeSAL could save about 60% over FuSAL, and 80% over random sampling. Having high confidence was also important because it could save the model from making errors in the early stages.

Related papers