The focus paper introduces a task called tense sense disambiguation, where given a concrete tense syntactic form in a sentence, the objective is to select the correct sense among a given set of possible senses.
As an experiment, an English syntactic sense dictionary was compiled and then used to annotate 3000 sentences from the British National Corpus. A supervised learning TSD algorithm was developed that used basic, lexical, and POS Tag-based features. The algorithm also caters to the task structure by allowing the restriction of the possible labels to each concrete syntactic form. The classifier outperforms the MFS baseline in all three conditions when the ASF (abstract syntactic form) type is known, unknown, or given by a simple rule-based classifier.
So for this week, people read the following supplemental papers:
Weisi read about a prototype model for coarse-to-grain learning with regards to multi-class classification. The model was pipelined with sets of independent classifiers trained at each level and since it does not use the overlap in the confusion lists on certain labels, there is the possibility for a better estimation of the feature weights.
Dong read about experiments in combining classifiers using various techniques to improve performance with regards to WSD. Although the features or combinations themselves may not have been that unique, the end result of the paper showed improved performance from such an approach.
In somewhat of a contrast, Daniel read about novel syntactic features used to improve WSD performance. The paper uses supervised machine learning methods, decision lists, and AdaBoost, as well as added features for subcategorization frames and dependencies on words already present with some given sense. Thresholding was also used to trade recall for precision. In the end, the specific syntactic features along with AdaBoost performed the best, although AdaBoost has no effective thresholding mechanism.
Dhananjay read about imposing global constraints over local pairwise order decisions. One is transitivity and the other is time expression normalization. Transistivity uses Integer LP while time expression normalization normalizes everything to a single timeline. The results show 1-2% absolute increase in accuracy.
Brendan read an overview of construction grammar. A construction is a paring of a meaning and form. They require consideration of communicative functionality, meaning, and general cognitive constraints. There are some good examples illustrated in the link on Brendan's post. It seems the relevance of the theory and details in terms of using it or comparing it in some applicable manner are still unclear.
-Alan
Thursday, March 3, 2011
Pre-meeting commentary
I read Goldberg 2003, an overview of construction grammar. (In the paper they cite Goldberg's 1995 book).
http://www3.isrl.illinois.edu/~junwang4/langev/localcopy/pdf/goldberg03constructions.pdf
A construction is a "pairing of form and meaning." Words, multiwords, morphological features, etc. are all constructions. To understand them you have to think about communicative functionality, meaning, and general cognitive constraints.
That's kind of it. Some examples:
It's a very NLP-friendly view of language. They're like the all the little features you get in MT or other tasks.
It's kind of like lexical semantics, except for more than just words.
It's kind of dissatisfying, in that there's no theory how words combine into meaning.
It's kind of not much of a theory. I don't see how it explains or makes testable hypotheses or predictions. Or even how you would use it to help design a grammar or NLP features, even. I don't see how you could hope to compare it to LFG or CCG or HPSG or minimalism or what have you.
I see how it might have helped Reichart and Rappoport think about these things, or something. I dunno if I should bother reading the book though.
http://www3.isrl.illinois.edu/~junwang4/langev/localcopy/pdf/goldberg03constructions.pdf
A construction is a "pairing of form and meaning." Words, multiwords, morphological features, etc. are all constructions. To understand them you have to think about communicative functionality, meaning, and general cognitive constraints.
That's kind of it. Some examples:
It's a very NLP-friendly view of language. They're like the all the little features you get in MT or other tasks.
It's kind of like lexical semantics, except for more than just words.
It's kind of dissatisfying, in that there's no theory how words combine into meaning.
It's kind of not much of a theory. I don't see how it explains or makes testable hypotheses or predictions. Or even how you would use it to help design a grammar or NLP features, even. I don't see how you could hope to compare it to LFG or CCG or HPSG or minimalism or what have you.
I see how it might have helped Reichart and Rappoport think about these things, or something. I dunno if I should bother reading the book though.
Wednesday, March 2, 2011
This week, I read Jointly Combining Implicit Constraints Improves Temporal
Ordering by Nathanael Chambers and Dan Jurafsky. The paper proposes imposing
global constraints over the local pairwise order decisions. It proposes two
constraints (1) Transitivity and (2) Time expression normalization.
Transitivity. A soft classification (before, after and unknown) is done using
SVM and confidence scores are calculated for each pairwise events. The objective
function is maximized subject to the constraints that only one (before, after,
and unknown) is chosen. Transitivity is imposed over the connected components of
a digraph that r_1 + r_2 - r_3 \leq 0. Integer Linear Programming is used to
find the optimal solution. This however, didn't find any advantage over the
Timebank Corpus as the graph was very sparse and connected components were very
few. To eliminate this, they create a transitive closure.
The second global constraint that was imposed was normalizing the time
expressions to a single timeline. The document's publication date was considered
to be the current point in time, and phrases like "last month", "next Friday"
are normalized.
After using global as well as transitivity, the authors report 1-2% (absolute)
increase in accuracy
Ordering by Nathanael Chambers and Dan Jurafsky. The paper proposes imposing
global constraints over the local pairwise order decisions. It proposes two
constraints (1) Transitivity and (2) Time expression normalization.
Transitivity. A soft classification (before, after and unknown) is done using
SVM and confidence scores are calculated for each pairwise events. The objective
function is maximized subject to the constraints that only one (before, after,
and unknown) is chosen. Transitivity is imposed over the connected components of
a digraph that r_1 + r_2 - r_3 \leq 0. Integer Linear Programming is used to
find the optimal solution. This however, didn't find any advantage over the
Timebank Corpus as the graph was very sparse and connected components were very
few. To eliminate this, they create a transitive closure.
The second global constraint that was imposed was normalizing the time
expressions to a single timeline. The document's publication date was considered
to be the current point in time, and phrases like "last month", "next Friday"
are normalized.
After using global as well as transitivity, the authors report 1-2% (absolute)
increase in accuracy
Pre-meeting commentary for 3/3
Focus Paper: Tense Sense Disambiguation: a New Syntactic Polysemy Task. Roi Reichart and Ari Rappoport. EMNLP 2010
Additional Reading: Syntactic features for high precision Word Sense Disambiguation. David Martinez et al. COLING 2002
This paper explores novel syntactic features for use in improving precision in Word Sense Disambiguation. The authors use existing supervised machine learning methods, Decision Lists, and AdaBoost. In addition to the standar n-gram type features already in use by earlier models of WSD, the authors add features for dependencies to specific words being present with a given word sense, as well as features for subcategorization frames. The authors then experimented with various types of thresholding in order to boost precision at the expense of recall.
When no thresholding, they found that the specific syntactic features helped precision dramatically, but the subcategorization features led to a higher F score, with the combination of the two being slightly better than the subcategorization features. They further find that AdaBoost works significantly better than Decision Lists when syntactic features are taken into account, although not with the basic feature set. AdaBoost combined with syntactic features outperforms either method with only basic features.
The authors found that there was not an effective thresholding mechanism for AdaBoost, but with decision lists, by only using features with high confidences, it was possible to prune the results to 95% precision and 7% recall.
Additional Reading: Syntactic features for high precision Word Sense Disambiguation. David Martinez et al. COLING 2002
This paper explores novel syntactic features for use in improving precision in Word Sense Disambiguation. The authors use existing supervised machine learning methods, Decision Lists, and AdaBoost. In addition to the standar n-gram type features already in use by earlier models of WSD, the authors add features for dependencies to specific words being present with a given word sense, as well as features for subcategorization frames. The authors then experimented with various types of thresholding in order to boost precision at the expense of recall.
When no thresholding, they found that the specific syntactic features helped precision dramatically, but the subcategorization features led to a higher F score, with the combination of the two being slightly better than the subcategorization features. They further find that AdaBoost works significantly better than Decision Lists when syntactic features are taken into account, although not with the basic feature set. AdaBoost combined with syntactic features outperforms either method with only basic features.
The authors found that there was not an effective thresholding mechanism for AdaBoost, but with decision lists, by only using features with high confidences, it was possible to prune the results to 95% precision and 7% recall.
Pre-meeting Review for 3/2/11, Alan Zhu
To recap I read "Latent features in automatic tense translation between Chinese and English."
The paper is a very basic introduction to the difficulties inherent in translating Chinese verbs to English, particularly with regards to the task of determining tense. The paper pushes the idea that the deeper features used by humans when translating needs to be identified and also should be the focus in terms of creating more advanced automatic extraction methods. As a method of comparison and evaluation, tense classifiers with latent features are demonstrated to have better performance than those only using surface features. The paper seems to promote a more human-based cognition approach to the problem and NLP in general, targeting the information that currently remains difficult to extract.
I feel Chinese may have just been chosen to demonstrated maybe the more interesting or extremal case of the tense classification problem. The basic feature space explored includes surface features, latent features, telicity and punctuality features, and temporal ordering features. It was fairly comfortable to understand the basic ideas here due to my relative fluency in Chinese, and I won't go into the details on this blog post.
The paper describes experiments using both CRF (conditional random fields) learning experiments and classification tree learning experiments. The classifiers are trained on both surface features and latent features separately and then together. Both yield similar accuracy and perform generally 15% better than baseline systems. I feel like a large part of the paper is to maybe reorient focus on extracting latent features as the authors leverage the difficulty of the task as the key to advancing our current extraction methods.
I will write up the pre-meeting summary before class tomorrow.
-Alan
The paper is a very basic introduction to the difficulties inherent in translating Chinese verbs to English, particularly with regards to the task of determining tense. The paper pushes the idea that the deeper features used by humans when translating needs to be identified and also should be the focus in terms of creating more advanced automatic extraction methods. As a method of comparison and evaluation, tense classifiers with latent features are demonstrated to have better performance than those only using surface features. The paper seems to promote a more human-based cognition approach to the problem and NLP in general, targeting the information that currently remains difficult to extract.
I feel Chinese may have just been chosen to demonstrated maybe the more interesting or extremal case of the tense classification problem. The basic feature space explored includes surface features, latent features, telicity and punctuality features, and temporal ordering features. It was fairly comfortable to understand the basic ideas here due to my relative fluency in Chinese, and I won't go into the details on this blog post.
The paper describes experiments using both CRF (conditional random fields) learning experiments and classification tree learning experiments. The classifiers are trained on both surface features and latent features separately and then together. Both yield similar accuracy and perform generally 15% better than baseline systems. I feel like a large part of the paper is to maybe reorient focus on extracting latent features as the authors leverage the difficulty of the task as the key to advancing our current extraction methods.
I will write up the pre-meeting summary before class tomorrow.
-Alan
Pre-meeting Dong
Pre-meeting (Dong Nguyen).
Related paper: Modeling Consensus: Classifier Combination for Word Sense Disambiguation, Radu Florian and David Yarowksy
Focus paper: Tense Sense Disambiguation: a New Syntactic Polysemy Task, Roi Reichart and Ari Rappoport
The related paper by Florian and Yarowsky describe experiments with different methods of combining classifiers to improve the performance on the word sense disambiguation task. They experimented with 6 different classifiers, and different methods of combining them (weighted average of posterior probability, combining based on order statistics and voting). Their features included bigrams, trigrams, BOW, and syntactic features. They showed that high performance gains could be gained by combining them. Their final approach ('stacking'), was a combination of different combinations of classifiers.
I think the main contribution of the paper was combining classifiers on WSD disambiguation. It doesn't seem their features itself are very innovative or was their focus. Furthermore, it's not clear how innovative the used classifier combinations were in general (not restricted to WSD).
The focus paper was interesting, but because I'm not familiar with some of the approaches their work builds on (sequential model, SNOW), some parts were not totally clear to me. Also, I found some of their defined senses seem to be a bit strange (i.e. 'when describing the content of a book' seems to be very specific).
Related paper: Modeling Consensus: Classifier Combination for Word Sense Disambiguation, Radu Florian and David Yarowksy
Focus paper: Tense Sense Disambiguation: a New Syntactic Polysemy Task, Roi Reichart and Ari Rappoport
The related paper by Florian and Yarowsky describe experiments with different methods of combining classifiers to improve the performance on the word sense disambiguation task. They experimented with 6 different classifiers, and different methods of combining them (weighted average of posterior probability, combining based on order statistics and voting). Their features included bigrams, trigrams, BOW, and syntactic features. They showed that high performance gains could be gained by combining them. Their final approach ('stacking'), was a combination of different combinations of classifiers.
I think the main contribution of the paper was combining classifiers on WSD disambiguation. It doesn't seem their features itself are very innovative or was their focus. Furthermore, it's not clear how innovative the used classifier combinations were in general (not restricted to WSD).
The focus paper was interesting, but because I'm not familiar with some of the approaches their work builds on (sequential model, SNOW), some parts were not totally clear to me. Also, I found some of their defined senses seem to be a bit strange (i.e. 'when describing the content of a book' seems to be very specific).
Pre-meeting Post from Weisi Duan
I have read the paper “A Sequential Model for Multi-Class Classification” by Yair Even-Zohar and Dan Roth. The paper disusses a prototype model for coarse-to-grain learning, but in the domain of multi-class classification, instead of structured prediction. The model is a pipelined model with a sequence of classifiers covering different feature space and output space, and the output space of the previous classifier is pruned by thresholding to generate the output space of the next classifier. This seems to suggest an explosion of classifiers if not well engineered. During training, the classifiers are trained based the pruned output space of the previous classifier and instances relevant to the pruned output space. The authors provide a proof on bigger output space induce more error and smaller output space reduce the training error. The idea gives the feeling that it basically training a set of independent classifier at each level, and does not take advantage of the overlap of the confusion lists on certain labels, which could result in better estimation for weights of the features f(x, overlapped_label). For example, in WSD, we engineer the features to reflect only semantic correlation, and not bond to the target words, in which case we would have one single classifier instead of multiple classifiers, with one for each target word respectively, and the features would be better estimated because of removal of the partition of training examples enforced by the target words. For the focus paper, I am curious about the methodology they used to generated the senses except for time constraints.
Subscribe to:
Posts (Atom)