The paper I mentioned on metaphors in sign language will be the subject of discussion for Monday's Linguistics Reading Group (which meets at 2:30 in GHC 7501). Anyone who's interested is welcome to participate.- Nathan
Thursday, January 27, 2011
Metaphor Comics
I can't resist sharing Dinosaur Comics' take on conceptual metaphor (below) and metaphor-related idioms [1], [2]:
The paper I mentioned on metaphors in sign language will be the subject of discussion for Monday's Linguistics Reading Group (which meets at 2:30 in GHC 7501). Anyone who's interested is welcome to participate.
- Nathan
The paper I mentioned on metaphors in sign language will be the subject of discussion for Monday's Linguistics Reading Group (which meets at 2:30 in GHC 7501). Anyone who's interested is welcome to participate.- Nathan
Reading for 2/3/11: Haghighi and Klein, 2010
An Entity-Level Approach to Information Extraction
Request (new!): When you post to the blog, please include:
Author: Aria Haghighi and Dan Klein
Venue: ACL 2010
Leader: Dhananjay
Request (new!): When you post to the blog, please include:
- Your name (plus "leader") if you are leading the discussion
- Which focus paper this post relates to
- Whether this is the pre-meeting review or the post-meeting summary
Reminders:
- Leave a comment on this post (non-anonymously) giving the details of the related paper you will read (include a URL), by Monday, January 31.
- Post your commentary (a paragraph) as a new blog post, by Wednesday, February 2.
Summary of week 2 commentary
As the focus paper this week was a survey, everyone chose to read papers detailing one of the methods discussed in the focus paper. With one exception, the papers people chose to read either focused on metaphor detection or metaphor interpretation.
Dani, Alan, Dhananjay and I read papers on metaphor detection.
Dani's paper, Metaphor Identification Using Verb and Noun Clustering, combines a small amount of seed knowledge in the form of source-target domain mappings with word clustering in order to generalize those mappings. The clustering is done using parse information and a spectral clustering algorithm. To evaluate, they sampled randomly from the output of their system, and had human annotators judge the sampled sentences, obtaining a precision of .79.
Alan's paper, Comparing Semantic Role Labeling with Typed Dependency Parsing in Computational Metaphor Identification, focused on a slightly different task: finding patterns in text that commonly indicate the use of metaphor. The paper found that semantic role labels are slightly more useful than typed dependency arcs for extracting semantic relations from text, but overall, the paper mostly discussed the problem instead of their solution.
Dhananjay read the paper Catching Metaphors, which used a maximum entropy classifier to detect the metaphorical usage of verbs. The features used in the model were the prior belief of each verb being used metaphorically, and the type of the verb's arguments. The paper used WSJ data that they annotated. A high accuracy of 96.98% accuracy is reported, although this weakened by the fact that over 90% of the verbs that were annotated were marked as metaphorical.
I read the paper Hunting Elusive Metaphors Using Lexical Resources. This paper looked at a wider range of phenomena than other metaphor detection papers, nouns, verbs, and adjectives, but used relatively simplistic techniques. In order to discover IsA metaphors, they simply checked to see if the first noun was a hyponym of the second, using WordNet. For verb and adjective metaphors, they used a method based on computing the frequency of a noun's hyponyms occurring as arguments of the predicate. Their evaluation was not well explained, and they had unimpressive results.
Next, Matt, Michael, Dong, and Weisi read papers that looked at metaphor interpretation.
Matt and Michael read the paper
Overall, the topic of metaphor in NLP seems to suffer from a lack of a good definition, and no standardization of evaluation. None of the papers present results that are comparable to any of the others, so it is hard to say conclusively what sorts of techniques are preferable.
Dani, Alan, Dhananjay and I read papers on metaphor detection.
Dani's paper, Metaphor Identification Using Verb and Noun Clustering, combines a small amount of seed knowledge in the form of source-target domain mappings with word clustering in order to generalize those mappings. The clustering is done using parse information and a spectral clustering algorithm. To evaluate, they sampled randomly from the output of their system, and had human annotators judge the sampled sentences, obtaining a precision of .79.
Alan's paper, Comparing Semantic Role Labeling with Typed Dependency Parsing in Computational Metaphor Identification, focused on a slightly different task: finding patterns in text that commonly indicate the use of metaphor. The paper found that semantic role labels are slightly more useful than typed dependency arcs for extracting semantic relations from text, but overall, the paper mostly discussed the problem instead of their solution.
Dhananjay read the paper Catching Metaphors, which used a maximum entropy classifier to detect the metaphorical usage of verbs. The features used in the model were the prior belief of each verb being used metaphorically, and the type of the verb's arguments. The paper used WSJ data that they annotated. A high accuracy of 96.98% accuracy is reported, although this weakened by the fact that over 90% of the verbs that were annotated were marked as metaphorical.
I read the paper Hunting Elusive Metaphors Using Lexical Resources. This paper looked at a wider range of phenomena than other metaphor detection papers, nouns, verbs, and adjectives, but used relatively simplistic techniques. In order to discover IsA metaphors, they simply checked to see if the first noun was a hyponym of the second, using WordNet. For verb and adjective metaphors, they used a method based on computing the frequency of a noun's hyponyms occurring as arguments of the predicate. Their evaluation was not well explained, and they had unimpressive results.
Next, Matt, Michael, Dong, and Weisi read papers that looked at metaphor interpretation.
Matt and Michael read the paper
A Fluid Knowledge Representation for Understanding and Generating Creative Metaphors. This paper automatically creates a knowledge base from WordNet and uses that knowledge to find semantic links between seemingly unrelated nouns. They find facts of the forms is_ADJ:NOUN and VERBs:NOUN by looking at parsed dictionary entries. They then create a graph of nouns, with links between nouns that have very similar facts. Connected nouns in the graph are then considered to be semantically related. The paper does not address in detail the problem of using this information for the task of metaphor interpretation.
Dong and Weisi's paper, Automatic Metaphor Interpretation as a Paraphrasing Task, addressed the problem of finding literal paraphrases of metaphorical verbs. They use a variety of methods to obtain and filter a list of possible paraphrases using WordNet similarity, likelihood given the context, and a selectional preference measure. They hand-annotated a set of sentences with a ranked list of possible verb paraphrases, and evaluated their system on 1st choice accuracy and mean reciprocal rank, getting an accuracy of .81. Weisi makes a comparison of this task to word sense disambiguation, noting that is much easier, since the problem of detecting metaphor is already taken care of.
Finally, Brendan read a paper about analogical reasoning, A Logical Approach to Reasoning by Analogy. This paper is not about metaphor, but rather about giving a precise account of reasoning by analogy. The paper gives a formal definition of reasoning by analogy, then generalizes it and discusses an implementation of it in a logical programming language. Brendan discusses how this might be useful in metaphor interpretation, and how the task of metaphor detection is not particularly interesting by itself.Dong and Weisi's paper, Automatic Metaphor Interpretation as a Paraphrasing Task, addressed the problem of finding literal paraphrases of metaphorical verbs. They use a variety of methods to obtain and filter a list of possible paraphrases using WordNet similarity, likelihood given the context, and a selectional preference measure. They hand-annotated a set of sentences with a ranked list of possible verb paraphrases, and evaluated their system on 1st choice accuracy and mean reciprocal rank, getting an accuracy of .81. Weisi makes a comparison of this task to word sense disambiguation, noting that is much easier, since the problem of detecting metaphor is already taken care of.
Overall, the topic of metaphor in NLP seems to suffer from a lack of a good definition, and no standardization of evaluation. None of the papers present results that are comparable to any of the others, so it is hard to say conclusively what sorts of techniques are preferable.
Wednesday, January 26, 2011
Comments for week 2
I read the paper:
T. Veale and Y. Hao. 2008. A fluid knowledge representation for understanding and generating creative metaphors. In Proceedings of COLING 2008, pages 945–952, Manchester, UK.
The goal of this paper is metaphor interpretation and generation (ambitious!). Michael gives a good overview of their approach which I would breakdown into 3 steps:
1. Extract facts
2. Link facts
3. Use knowledge representation to interpret metaphors
#1 seems straightforward and they accomplish it using WordNet and the web. They also have empirical results demonstrating the quality of their facts.
#2 seems more difficult and they give some small amount of details of how they identify closely related facts using semantic relations in WordNet. #3 they mostly explain by examples linking together two seemingly unrelated nouns like "Pope" and "Don (Crime Father)". It seems to me that the algorithm can be thought of as constructing a graph using some heuristic rules and then finding a path (any path?) through the graph from A to B. Its unclear to me how this could be used to interpret a new metaphor and the authors don't seem to address this directly. This also seems to contrast with earlier cited work which is also called a slipnet and is referred to as a "probabilistic network". As far as I can tell, there is nothing stochastic about their approach. I briefly endeavored to read the earlier work (Hofstadter, 1994) but failed due to the fact that it is 80 pages.
They also don't have any way of measuring the quality of their interpretations which, in fairness, seems like a difficult task.
Matt
T. Veale and Y. Hao. 2008. A fluid knowledge representation for understanding and generating creative metaphors. In Proceedings of COLING 2008, pages 945–952, Manchester, UK.
The goal of this paper is metaphor interpretation and generation (ambitious!). Michael gives a good overview of their approach which I would breakdown into 3 steps:
1. Extract facts
2. Link facts
3. Use knowledge representation to interpret metaphors
#1 seems straightforward and they accomplish it using WordNet and the web. They also have empirical results demonstrating the quality of their facts.
#2 seems more difficult and they give some small amount of details of how they identify closely related facts using semantic relations in WordNet. #3 they mostly explain by examples linking together two seemingly unrelated nouns like "Pope" and "Don (Crime Father)". It seems to me that the algorithm can be thought of as constructing a graph using some heuristic rules and then finding a path (any path?) through the graph from A to B. Its unclear to me how this could be used to interpret a new metaphor and the authors don't seem to address this directly. This also seems to contrast with earlier cited work which is also called a slipnet and is referred to as a "probabilistic network". As far as I can tell, there is nothing stochastic about their approach. I briefly endeavored to read the earlier work (Hofstadter, 1994) but failed due to the fact that it is 80 pages.
They also don't have any way of measuring the quality of their interpretations which, in fairness, seems like a difficult task.
Matt
Related Paper - Jan 27
I read the paper :
E. Shutova, L. Sun and A. Korhonen. 2010. Metaphor Identification Using Verb and Noun Clustering. In Proceedings of COLING 2010, Beijing, China.
http://www.cl.cam.ac.uk/~es407/papers/Coling10.pdf
The paper describes a word clustering approach to metaphor identification. Their decision to use word clustering is based on hypothesis that target concepts associated with a source concept appear in similar lexico-syntactic environments, and clustering will capture this relatedness by association. The method starts with a small set of seeds of source-target domain mappings, extracts rich features from a shallow parser, and uses spectral method to perform noun and verb clustering. The resulting noun clusters are considered as target concepts in the same source domain, and the resulting verb clusters are considered as source domain lexicon.
As for the results, they were able to get some nice metaphors that represent broad semantic classes such as {swallow anger, hurl comment, spark enthusiasm, etc.} from seeds {stir excitement, throw remark, cast doubt}, which the WordNet-based approach (baseline) cannot acquire. They evaluated the methods using precision, and got 0.79 (baseline 0.44). I don't think these numbers are convincing though since they randomly sampled sentences annotated by the systems and asked five human annotators to judge, but they did not report the size of the sample (or maybe I missed it?). Also, though there is no large annotated corpus for metaphor identification, it would be nice if they had reported recall on smaller data just to get an idea of the coverage of the method.
E. Shutova, L. Sun and A. Korhonen. 2010. Metaphor Identification Using Verb and Noun Clustering. In Proceedings of COLING 2010, Beijing, China.
http://www.cl.cam.ac.uk/~es407/papers/Coling10.pdf
The paper describes a word clustering approach to metaphor identification. Their decision to use word clustering is based on hypothesis that target concepts associated with a source concept appear in similar lexico-syntactic environments, and clustering will capture this relatedness by association. The method starts with a small set of seeds of source-target domain mappings, extracts rich features from a shallow parser, and uses spectral method to perform noun and verb clustering. The resulting noun clusters are considered as target concepts in the same source domain, and the resulting verb clusters are considered as source domain lexicon.
As for the results, they were able to get some nice metaphors that represent broad semantic classes such as {swallow anger, hurl comment, spark enthusiasm, etc.} from seeds {stir excitement, throw remark, cast doubt}, which the WordNet-based approach (baseline) cannot acquire. They evaluated the methods using precision, and got 0.79 (baseline 0.44). I don't think these numbers are convincing though since they randomly sampled sentences annotated by the systems and asked five human annotators to judge, but they did not report the size of the sample (or maybe I missed it?). Also, though there is no large annotated corpus for metaphor identification, it would be nice if they had reported recall on smaller data just to get an idea of the coverage of the method.
Readings for Jan. 27
As a refresher I read,
"Comparing Semantic Role Labeling with Typed Dependency Parsing in
Computational Metaphor Identification"
First off, the focus paper was slightly different than what I expected. Not that it was a bad read, but it seemed more like a general history of solving the metaphor problem and the current state of things. A lot easier to read than normal papers, and it presents some really interesting work, namely MIDAS, which I thought presented a very smart approach to the problem. Generally speaking, I feel like aggregating a huge repository of hand-generated metaphors isn't really the most elegant solution to the problem, which I guess not only emphasizes the difficulty of the problem itself, but the question of whether there's some slick solution to "successfully" triumphing over metaphor detection and resolution.
The additional paper I read attempted to use semantic role labeling in pace of typed dependency parsing to improve CMI, which is an approach that aims to identify patterns that indicate metaphors, and not worry so much about actually identifying each metaphor. I suppose you could call it a higher-order version of metaphor detection. Unfortunately, the paper doesn't have much to report in terms of result. Much of the paper was spent asking questions and giving background rather than actually talking about the significance of the research. It turns out that semantic role labeling proves slightly more effective in extracting relationships that have more semantic importance, but it's a double edged sword in that the granularity may be too fine to prove of effective use for input into other systems.
The additional paper I read attempted to use semantic role labeling in pace of typed dependency parsing to improve CMI, which is an approach that aims to identify patterns that indicate metaphors, and not worry so much about actually identifying each metaphor. I suppose you could call it a higher-order version of metaphor detection. Unfortunately, the paper doesn't have much to report in terms of result. Much of the paper was spent asking questions and giving background rather than actually talking about the significance of the research. It turns out that semantic role labeling proves slightly more effective in extracting relationships that have more semantic importance, but it's a double edged sword in that the granularity may be too fine to prove of effective use for input into other systems.
-Alan
Commentary for Jan 27th
I read the paper Hunting Elusive Metaphors Using Lexical Resources. Saisuresh Krishnakumaran and Xiaojin Zhu. 2007.
This paper focuses on just the task of metaphor identification, although they consider metaphors expressed by nouns, adjectives, and verbs, unlike much of the related work which looks at only nouns or only verbs. The authors first discuss various general challenges in the task. They cover context-sensitive metaphors, metaphors that require reference resolution to identify, and metaphors that are not identifiable using lexical semantics alone. The authors then restrict their attention to three forms of metaphor: noun1-IsA-noun2, verb-object, and adjective-noun. To determine if a given sentence is metaphorical, they parse the sentence using Klein and Manning's unlexicalized phrase-structure parser, then look for each form of metaphor separately. To find the first type, they use a the following simple heuristic: if two nouns are in an IsA relationship and the latter is not a hypernym of the former, the sentence is metaphorical. The second two types are both done using the same method. For a predicate-noun pair, a corpus is searched for every instance of that predicate, and the probability of each noun being its argument is computed. If neither the target noun nor any of its hyponyms has a high enough probability in this distribution, the sentence is marked as metaphorical.
The methods this paper uses are fairly crude, and they do not do a very good job of explaining their evaluation. They appear to use data that they annotated with WordNet in mind, which seems problematic, since they use WordNet as a resource when running on the test set. They do not report their accuracy clearly, but they appear to get F measures in the 55-65% range.
Subscribe to:
Posts (Atom)