At the end of class, we brought up the possibility of using cross-entropy to do text classification. If anyone is curious, here's an interesting paper that builds n-grams of the LETTERS (instead of the words):
http://citeseer.nj.nec.com/teahan00text.html
Amoung other tasks, they mention results comparable to the ones we saw in the presentation for the NewsGroup data. But this method has the benefit of not having to know the optimal vocabulary size in advance.

Is not stemming really what's violating the naive bayes assumption? To me, it seems like independence of words is a poor approximation to begin with, and the stemming issue wouldn't make a huge difference either way. But from a practical point of view, it's nicer to have a system that just takes training data "as is" (without extra preprocessing).

Does anyone else think the results aren't all that conclusive? They don't have that much training/testing data for starters. And the fact that they show test results over a range of vocab sizes makes me wonder "how would I know what vocab size to use in practice?". For some of the test results (Newsgroups and WebKB) there doesn't seem to be much of a difference between multinomial and bernoulli.

Edited by author 05-12-2003 07:33 PM
I wonder what N_subscript"is" represents versus the Nit Andrew mentioned in his query. Is this a misprint or across all the possible occurences?
Are the factorial part of their probability distributions? I don't have it directly infront of me.

Neil, they use a many-to-one correspondence of mixture components to class labels in the next paper "Text Classification from Labeled and Unlabeled Documents using EM." Basically, the mixture components correspond to sub-topics whereas the class labels correspond to more general topics (ie baseball vs. sports). look on Page 14 for their example.

Does anyone know the intuition behind the |d_i|! and N_it! in equation (5)? I usually see the probability of a document given a class as equation (5) without those two quantites.

I'm curious why they don't use stemming in their text processing. By not stemming they're violating the naive bayes assumption (independence of words). It seems trivial to apply a stemming algorithm, so there must be some (unstated) reason for avoiding it.

Also confusing to me is the sort of vague statement that one *could* make it so that there is not a one-to-one correspondence between class labels and the mixture model components. If the two don't correspond, what is the mixture? Could parts of one document belong to two different classes? I can't seem to get my head around what this represents.