Comments week 2

This week I have read the following related paper:

E. Shutova. 2010. Automatic Metaphor Interpretation as a Paraphrasing Task. In Proceedings of NAACL 2010, Los Angeles, USA.

http://www.cl.cam.ac.uk/~es407/papers/NAACL.pdf

This paper frames metaphor interpretation as a paraphrasing task. Given a metaphorical expression, the system returns a ranked list of literal substitutions. The author only focuses on single-word metaphors expressed by a verb. First paraphrases are ranked according to their likelihood in the context. Unrelated substitutions are then removed, by only keeping terms that are a hypernym or share a hypernym with the metaphor according to WordNet. A selectional preference measure is then used to filter out metaphors and rerank the paraphrases (how they did the reranking wasn't very clear to me). They showed in their evaluation that the last step increased performance a lot. I liked their approach, because the steps they performed are intuitive and relatively simple.

They evaluate their system in two different ways. By looking at the accuracy of the returned paraphrases ranked first, and at the MRR with a cutoff at rank 5. I think their performance was pretty good, with an accuracy of 0.81 when only looking at the first returned paraphrase. However, there are often multiple suitable substitutions for a metaphor (their annotators also had to list all suitable literal paraphrases they could come up with for the particular verb). It would have been interesting not only to look at which ranking the first correct paraphrase occurred, but also for example how many of the paraphrases in the top x were actually correct (accuracy instead of MRR).