Reading for 3/31/11: Ramage et al., ICWSM 2010

Characterizing Microblogs with Topic Models

Author:  Daniel Ramage, Susan Dumais, Dan Liebling

Venue:  ICWSM 2010

Leader:  Daniel Mills

Request:  When you post to the blog, please include:

1. Your name 
2. Which focus paper this post relates to
3. Whether this is the pre-meeting review or the post-meeting summary

Reminders:

• Leave a comment on this post (non-anonymously) giving the details of the related paper you will read (include a URL), by Monday, March 28.
• Post your commentary (a paragraph) as a new blog post, by Wednesday, March 30.

Post Meeting - Dhananjay

The meeting revolved mostly around evaluation of Topic Models. For most of the time, we discussed the role focus paper played (or the objective of the focus paper). The method of using the first few lines as the summary of the topic was challenged. It was concluded that the reader should have access to the complete article, should it be wanted.

The behavior of the evaluation for models that generate n-grams was put forward.. It is interesting, how the word intrusion task would perform in such cases, for example, a topic that rates high on a bigram like New York, but rates low for York.

The final point of discussion was what role Topic models play. They are not useful as a pre-processing or a decision making criterion for any other tasks. However, it was accepted that with the advent of topic models, the research in generative graphical models matured

Post meeting Dong

Post-meeting.
Focus paper: Reading tea leaves: How humans interpret topic models

Yesterdays meeting focused mainly on ways of evaluating topic models.
For the focus paper, the overall opinion was that the results weren't very strong. More details regarding the Mechanical Turk setup/results
would have been nice, as well as a comparison with somewhat less similar models, such as LSA. Furthermore, it would have been nice if they have connected topic models more with human cognitive processes.
During the meeting we also talked about calculating perplexity. There seems to be multiple ways of calculating this for topic models.

The author of my related paper seems to be working on presenting topic to humans. His recent publications contain a lot of papers regarding topics such as visualization of topics, external evaluation, topic labeling etc. Thus his website (http://www.ics.uci.edu/~newman/) might be a good starting point to search if you're interested in this.

A paper that was mentioned during the meeting was Not-So-Latent Dirichlet Allocation: Collapsed Gibbs Sampling Using Human Judgments by Jonathan
Chang (http://www.aclweb.org/anthology/W/W10/W10-0720.pdf), where humans simulate the sampling step of Gibbs sampling and construct a topic model. I haven't read it yet, but it looks very interesting.

Post meeting summary by Weisi Duan

The meeting today concentrated on issues involving the evaluation of the topic models. The discussion has been mainly revolved around the focus paper, which presents tasks to demonstrate the semantics of the topics obtained from the topic models. The inter-rater agreement, if is there, would be nice, but as suggested by Noah, the statistical significance of the results is still valid. For automatic evaluation, Dong suggested the measures in related paper does not address the issue strongly. Daniel read a paper about various statistical measures for topic models and suggested the comparison does not make enough sense. Alan read a paper about applying topic models to polylingual case to help machine translation, and it is suggested that machine translation is not helped a lot by topic models so far. Dipanjan asked about current work of applying the topic models applied to structured prediction, and my reading turned out to a good fit, since the model is an integrated bayesian model for WSD. Dhananjay read a paper about the adapting the PLSA over time. Brendan asked about how exactly the perplexity on the test data is calculated, in other words how to fix the document distributions. People have pointed out various points that are confusing in focus paper (eg. Alan pointed out about showing the topic results to the raters and claiming there is no bias does not make sense) and can be improved. It would be great if the results turned out to meet people’s expectations, such as CTM performed the best in the tasks and so on. It is suggested a lot of work could be done to improve the focus paper.

Pre-meeting commentry - Dhananjay

Focus Paper : Reading tea leaves: How humans interpret topic models. Chang et al. NIPS 2009
Related Paper : Topic Evolution in a Stream of Documents. A e Gohr et al. SIAM: 2009 Data Mining
(http://www.siam.org/proceedings/datamining/2009/dm09_078_gohra.pdf)

The related paper addresses the problem of changing nature of document collections. It tries to adapt the feature space and underlying document model. The idea is to generate a PLSA model for every fixed length time window. The paper provides an adaptive model for computing PLSA as a substitute to relearning for every window.

The PLSA model is parameterized by documents, words and topics. For every time window, the words and documents of the previous time window are discarded and new words are introduced. EM is used for inference. The MAP estimates of the previous window, remain as the current iteration estimates for the current window. For the current time, the documents that came before the time window are discarded, and so is the vocabulary that is not present in the current window. New words are "folded in" and the model is inferred again.

For evaluation, the authors used ACM-SIGIR conferences from 2000-2007. They compared their model with independent PLSA for a window for every time. The difference was the initialization. While the current model used the MAP from the previous computation, the independent PLSA was initialized randomly. The comparison was carried out for two windows - 1 year, and 2 years, and using the natural order and random order of documents. The average perplexity (50 iterations over k = 1, 2 4, ..., 128) for adaptive PLSA comes about 5% less than the independent PLSA.

Pre-meeting Commentary - Alan

Name: Alan

Related Paper: Polylingual Topic Models, Mimno et al. EMNLP 2009Name: Alan

Focus Paper: Reading tea leaves: How humans interpret topic models. Chang et al. NIPS 2009

The related paper introduces a polylingual topic model that discovers topics aligned across multiple languages. They look at documents that are translated word-for-word via the EuroParl corpus as well as those that are not directly translated but very likely to be about similar concepts (wikipedia articles in various languages). The aim is to evaluate whether PLTM can accurately infer topics over direct translations, infer similarities between vocabularies in different languages, and detect differences in topic emphasis between languages.

PLTM is an extension of latent Dirichlet allocation (LDA) and topic assignments can be inferred using Gibbs sampling. The generalization ability of the model is based on the probability of previously unseen held-out document given posterior estimates. To evaluate the possibility of using PLTM for adapting machine translation systems, the authors also measure the ability of the model to align documents in one language with their translations in another language.

Pre-meeting summary by Weisi Duan

Because I have one meeting and one class consecutively Thursday afternoon before the meeting, I may not have enough time to cover posts submitted after 1:00pm. However, I will try my best to update and cover all the new posts in time.

For this week’s focus paper, people have read different papers regarding the interpretation and application of the topics generated by the topic models.

Daniel has read the paper “Evaluation Methods for Topic Models.” by Wallach et al. ICML 2009. The paper explores various methods for evaluating LDA. The authors conduct the experiments in two settings: held-out documents, in which entire documents are held out, and document completion, in which only the latter half of each document is held out. The methods includes harmonic mean sampling, annealing importance sampling, estimated theta, Chib-style estimation, and left-to-right evaluation. The evaluation methods are compared by checking which method assigns higher probabilities to the held-out data, as well as the variance and computational complexity. The method for comparing evaluation methods does not seem well-justified, as noted by Daniel.

Dong read the paper “Automatic Evaluation of Topic Coherence” by Newman et al NAACL 2010, which is an extension of the focus paper. The paper explores 15 different automatic measures utilizing WordNet, Wiki, Google and a bunch of external methods. Dong suggests that more error analysis could be given and how well the method generalizes over domains is not known. The comparison on the ratings are also suggested to be not entirely convincing. About the focus paper, Dong suggests that the sensitivity of the parameters might affect the strength of their results.

I myself read the paper “A Topic Model for Word Sense Disambiguation” by Jordan Boyd-Graber, David Blei, and Xiaojin Zhu in EMNLP 2007. The paper is not exactly about the evaluation of topic models, but explores as an application of the semantic influence of the topics obtained by the topic models. The model of WSD is a packed model of P(topic | Corpus) and P(sense | topic). For the second model, the author uses WordNet-Walk which is model over all the possible paths on WordNet to a specific target word. The model is elegant in the sense that its components are modular. However, it does not work very well because of the structure of WordNet. There are also issues about estimation of P(sense | topic) where the path length is not paid enough attention. The perk of the whole model is that it is modular and can be integrated into bigger models. However, how well the approximate inference would work is not known. About the focus paper, I feel the authors could have provided the inter-rater agreement, because the raters are not very reliable.

Dhananjay read the paper “Topic Evolution in a Stream of Documents” by A e Gohr et al. in SIAM: 2009 Data Mining. The paper discusses about adapting topic models over time using PLSA. A PLSA model is obtained for each time period and for each new time period the estimates from the previous time period is used for intialisation. Comparison with randomly initialized PLSA model demonstrates a improvement of 5% in perplexity, as noted by Dhananjay.

Alan read the paper “Polylingual Topic Models” by Mimno et al. in EMNLP 2009. The paper is aimed at exploring the LDA to infer topics over direct translation, capture similarities between vocabularies in different languages, and detect differences in topic emphasis between languages. The evaluation is conducted on held out data using likelihood.To evaluate the possibility of using it for adapting machine translation systems, the authors also measure the ability of the model to align documents in one language with their translations in another language, as noted by Alan.