Jansche 2002 Information extraction from voicemail transcripts
Contents
Citation
Jansche, M. and Abney, S. 2002. Information Extraction from Voicemail Transcripts. In Proceedings of ACL-EMNLP.
Online version
An online version of this paper is available [1].
Summary
This paper introduces a simple yet effective way to extract Caller Phrase/Name and Phone Number from the voicemail transcripts. The author presents the detailed empirical results and statistics drawn from the corpus.
Key Contributions
The paper presents the following key findings between the trade-off of heuristic based simple classifier and rich-feature based sophisticated classifier
- The second one might have serious over fitting problems and prone to errors in unseen values of attributes (for example in ASR outputs)
- The first one exploits both the linguistics intuitions and empirical distributions thus is able to rely on strong heuristics and simple classifier
Mathematical Definition of MeMMs
In Maximum Entropy Markov models (MeMMs), the HMM transition and observation functions are replaced by a single function that provides the probability of the current state given the previous state and the current observation. In contrast to HMMs, in which the current observation only depends on the current state, the current observation in an MEMM may also depend on the previous state. In other words, the observations can be thought of as being associated with state transitions rather than with states. Moreover, for each state, such a transition function is given by a distinct Maxent model.
The paper illustrates a modification of the HMM Viterbi inference algorithm to perform inference for MeMMs. It also provides a Generalized Iterative Scaling (GIS) algorithm to train MeMMs, which is guaranteed to converge to a global maximum as long as there is at most one state per label.
For cases where training label sequences are unknown or ambiguous, the paper also gives a modification of the standard Baum-Welch training algorithm for HMMs.
Variations of MeMMs
The paper produces several variations of the basic MeMM architecture explained above:
- Factored state representation
To deal with data sparseness problem in standard MeMMs (due to transition parameters), one can avoid having S different transition functions (one for each state), and just maintain one function, which uses information about the previous state as features. This reduces the expressive power of the model but allows sharing of information across states and alleviates sparseness problems.
- Observations in states instead of transitions
Rather than combining transition and emission parameters into a single function, one could represent the transition probabilities as a standard multinomial, and P(S|O) using a Maxent model. This may also help with sparseness.
- Environmental model for reinforcement learning
The transition function can also include an action, resulting in a model suitable for representing the environment of a reinforcement agent.
Experiments
The authors trained 4 different types of models to classify lines from Usenet files into one of 4 categories: head, question, answer and tail. They used a set of 24 boolean features. The types of models they trained were: ME-Stateless (non-sequential Maxent), TokenHMM (a standard 4-state fully connected HMM), FeatureHMM (an HMM where the lines i.e. obsevations were replaced by their corresponding features), and the MeMM model described above. They found that the MeMM outperformed the other approaches.
Related papers
The Huang et al., 2001 paper discussed a very similar problem but rather with a traditional perspective, it studied three approaches: hand-crafted rules, grammatical inference of sub-sequential transducers and log-linear classifier with bi-gram and tri-gram features, which is essentially the same as in Ratnaparkhi, 1996 paper on Maxent POS tagging.
