View source for Ling, X. and Weld, D. Temporal Information Extraction. AAAI-10

== Reviews of this paper ==
{{#ask: [[reviewed paper::Ling,_X._and_Weld,_D._Temporal_Information_Extraction._AAAI-10]] | ?reviewer}}

== Citation ==

{{MyCiteconference | booktitle = Proceedings of the Twenty Fifth National Conference on Artificial Intelligence| coauthors = D.S. Weld| conference = AAAI| date = 2010| first = X.| last = Ling| title = Temporal Information Extraction| http://ai.cs.washington.edu/www/media/papers/tmpCKKIcN.pdf }}

== Online version ==

This [[category::paper]] is available online [http://ai.cs.washington.edu/www/media/papers/tmpCKKIcN.pdf].

== Summary ==

This [[Category::paper]] addresses the problem of  [[AddressesProblem::Temporal ordering]]. Unlike other works that focus on time-agnostic extraction of facts, this paper presents '''TIE''', a pipeline system that tries to extract as many facts from text while also inducing as much temporal information as possible. Temporal relations between events and times in a sentence are identified while transitivity is enforced to bind the start and ending time points for each event. The paper also presents ''temporal entropy'' as a way to evaluate recall of [[AddressesProblem::temporal information extraction]] systems.

== Brief description of the method ==

The objective of this paper is given a sentence, output (1) a maximal set of temporal elements that are either events (e.g. "officers were dispatched") or temporal references (e.g. "1999") and (2) the tightest set of temporal constraints as directly implied by the text. 

The constraints are linear inequalities of the form <math>p_1 \le p_2 + d \,\!</math> where <math>d \,\!</math> is a duration (often zero) and <math>p_1 \,\!</math> and <math>p_2 \,\!</math> denotes either the beginning (<math>^\triangleleft e \,\!</math>) or ending time point (<math>e^\triangleright \,\!</math>) of a temporal element <math>e \,\!</math>.

For example, the sentence 'Steve Jobs revealed the iPhone in 2007' might produce these constraints:
:<math>^\triangleleft Year-2007 \le ^\triangleleft Reveal(Jobs, iPhone) \,\! </math>
:<math>Reveal(Jobs, iPhone)^\triangleright \le Year-2007^\triangleright \,\! </math> 


The paper introduces a pipeline system called '''TIE''' with these components:
* '''Preprocessing''': (1) parse the sentence using a syntactic parser ([http://nlp.stanford.edu/software/lex-parser.shtml the Stanford parser]), (2) detect semantic roles for verbs in the sentence using [http://acl.ldc.upenn.edu/W/W05/W05-06.pdf#page=195 SRL], (3) use [http://timeml.org/site/tarsqi/index.html TARSQI] to find temporal elements (events and times) in the sentence, (4) generate features for each temporal element as well as between elements. 
:The features include descriptional features (tense, grounded time values, etc) generated for each element by TARSQI and syntactic features (dependency features and SRL features). 
* '''Classification''': use probabilistic model trained on [[UsesDataset::TimeBank Corpus]] combined with transitivity rules to classify each pair of points (<math>^\triangleleft e \,\!</math> or <math>e^\triangleright \,\!</math>) of elements the point-wise relation.
:[[UsesMethod::Markov Logic Networks]] is used to make predictions. Formula weights are learned from the TimeBank data. Some of the formula templates are as follows:

:<math>after(p,q) \and after(q,r) \to after(p,r) \,\!</math>
:<math>srl\_after(p,q) \to after(p,q) \,\!</math>

:where the first formula is transitivity and the second formula integrates the temporal information provided by SRL.

== Experimental Result ==

The experiment is conducted on sentences extracted from [http://www.cs.washington.edu/ai/iwp/tie.html Wikipedia] articles. The ground truth is constructed by asking human subjects to label the temporal relations between all pairs of elements by comparing their start and ending points. Each pair is labeled by at least two people with a third person to resolve any disagreement. 

Performance of '''TIE''' is measured using precision that measures the number of correct predictions over all predictions made in test data and ''temporal entropy'' (TE) that measures the number and "tightness" of the induced temporal bounds. A human or a system can give time bounds to the starting and end points of the element in the text. Temporal entropy of a time point is thus measured as the log of the length of the tightest bound produced for the time point. The lower the entropy, the better (tighter) is the bound. 

The precision and TE of '''TIE''' are compared to three other systems: (1) [http://aclweb.org/anthology/I/I08/I08-1054.pdf PASCA], a pattern-based extraction system for question answering, (2) [http://acl.ldc.upenn.edu/W/W05/W05-06.pdf#page=195 SRL], (3) [http://timeml.org/site/tarsqi/index.html TARSQI]. In terms of precision, '''TIE''' is able to extract more correct constraints at comparable precision to PASCA and SRL. In terms of TE, '''TIE''' has temporal entropy that is the closest to a human's skill. From the experiments, it is also observed that SRL feature improves the precision of '''TIE''' while transitivity feature improves the recall of the system. On [http://www.timeml.org/tempeval/ TempEval] task, '''TIE''' is able to achieve an accuracy of 0.695 which is comparable to that of the current state of the art, 0.716 ([[RelatedPaper::Yoshikawa 2009 jointly identifying temporal relations with markov logic |Yoshikawa et al. 2009]]).

== Discussion ==

This paper addresses the problem of [[AddressesProblem::temporal information extraction]] and [[AddressesProblem::Temporal ordering]] by using an assembly of existing systems in a pipeline called '''TIE''' to extract temporal elements and their features from text and [[UsesMethod::Markov Logic Networks]] to classify the temporal constraints between the starting or ending time points of these extracted elements. The system also proposes a measure called ''temporal entropy'' to measure the number and the tightness of the constraints produced. 

'''TIE''' is shown to outperform the other three approaches: PASCA, SRL, TARSQI in terms of precision and temporal entropy. 

Despite the good performance results, the contribution of this paper to the area of temporal ordering and temporal information extraction seems rather minimal as it simply uses an array of already existing systems. The idea of using Markov Logic Network for temporal ordering is also not new and it ([[RelatedPaper::Yoshikawa 2009 jointly identifying temporal relations with markov logic |Yoshikawa et al. 2009]]) even outperforms the system proposed in this paper in terms of accuracy. The main argument for the paper is that it predicts a different temporal ordering than that of other systems. Instead of classifying temporal relations between elements into a set of classes: BEFORE, AFTER, OVERLAP; this paper classifies the temporal relations between the elements' starting or ending time points. 

== Related papers ==

A related paper is [[RelatedPaper::Yoshikawa 2009 jointly identifying temporal relations with markov logic]] that also uses Markov Logic Networks to do a ''softer'' (non-deterministic) joint inference for temporal ordering of events and times. 

Another related paper is [[RelatedPaper::Denis and Muller, Predicting Globally-Coherent Temporal Structures from Texts via Endpoint Inference and Graph Decomposition, IJCAI 2011]] which, similar to this paper, attempts to classify all types of temporal relations (not just ''before''/''after'') in TimeBank Corpus by translating these temporal interval relations to their end points (starting and ending).