Should we standardize binary features too?

/m79: No. Standardizing binary features is unlikely to be beneficial. You would lose sparsity but not change the dynamic range much.

On a related note, how can we tell the statistical significance of a change in the expected value of a binary feature? Normally I would measure this in terms of standard deviation, but with binary values, the standard deviation is relatively high.

/m81: Standard deviation (SD) is still appropriate for evaluating the significance of a change in the average of a binary feature. The fact that the SD is high indicates that a large sample is needed.

It is a general fact that averages of 0/1 variables are noisy. This is an important reason why training an algorithm to minimize 0/1 error can be worse than training it to minimize something continuous, like MSE.

Edited by author 02-04-2010 06:29 AM
Dear all,
I have two questions.
1. How to analyze the space complexity experimentally?
2. How to confirm that my linear regression functions yield unbiased estimates of spending amounts? Could it be done by observing the error variance of the linear regression? If it is done by comparing the estimate value with the true value, I am not sure how to compute true average spending amount for each x.

/m83:

1. I don't think you need to say too much here. If in your implementation you needed to use more space than theoretically required, you can make a note of it. For example, since for loops are slow in MATLAB, you might find that it's faster to do some operation by first storing a lot of data in a matrix and then performing matrix algebra on it. This might require a bit more space than if you just use data "on the fly" with a for loop.

2. I think the simplest thing to do is to compute the average prediction of your linear regression model, and compare that with the true average spend amount of the data. If the two are roughly the same, then your model is returning an unbiased estimate.

After the quiz yesterday, Professor Elkan mentioned that the use of the term "informative" may have been misleading. I took it to mean that there was some true correlation between the feature and the output, opposed to an imagined correlation that is gathered from limited data. However, if this is the case, isn't overfitting possible regardless of whether or not the feature is informative?

/m85: Yes, overfitting is possible for informative features, as well as for uninformative features. By "informative" I did mean "some true correlation between the feature and the output", yes.

If a feature is completely uninformative, then any correlation found for it is overfitting. However, if a feature is informative, then a correlation discovered for it can be a combination of the truth and of overfitting.

Could the psychological analog for overfitting be "superstition"? -M


On Fri, Feb 5, 2010 at 3:00 PM, QT - Charles Elkan <
qtopic-43-NxEXMBcjigPp@quicktopic.com> wrote:

>
< replied-to message removed by QT >

Does matlab some how automatically recognize sparse matrices and display only the non-zero entries? That's what is seems like is doing when I look at slices of the data set, but just wanted to make sure, as I've not seen anything like it before. Thanks.

/m88: You're referring to the classic400 dataset, right? The MAT file has saved the matrix in sparse format, so that's why you are observing only nonzero entries when you take slices. This is standard MATLAB behaviour in my experience. AFAIK, MATLAB doesn't automatically convert a matrix to sparse format because of the potentially negative performance implications. So if you do
x = rand(10);
x(1:2, 1:2)

it will output a full 2x2 matrix of zeros. I hope that answers your question?

Yes, it does, thanks. So when using the data with the algorithm, we should use full(classic400) instead of the sparse version to get speedier results?

/m90: You certainly may store the data in full format; you just need to briefly justify your choice. A couple of things to consider are: (i) is it feasible to store the data in memory with the full format?, and (ii) in your implementation of Gibbs sampling, do you need to perform operations that operate only on the nonzero elements of the matrix? Will your inner loop be faster with the sparse or full format? As long as you demonstrate you have thought about these issues, you can use either format.

Spring 2010 CSE 291: Data mining and predictive analytics

This is the course I'll be teaching next quarter. It will have little overlap with 250B. Students are welcome to take either course, or both.

The data mining course will be more focused on application topics. It will require much less programming, and instead use the Rapidminer environment.

291 will meet just once per week, from 6:30pm to 9pm on Tuesdays, to accommodate participants with full time jobs. But graduate students are very welcome also.

Here is a paper that might be useful for the project: "Finding scientific topics" by Griffiths and Steyvers. It should aid in intuition about how LDA operates.

http://psiexp.ss.uci.edu/research/papers/sciencetopics.pdf

Quiz 5 grading scheme. Grading scheme for Quiz 5 follows. As always, please feel free to see me if you have any concerns about your quiz.

Part a.

0 pts: Answers True.

1 pt: Answers False, reasoning incorrect or elusive in details. e.g. "p(visit | x) will be less likely to overfit because we will have to spend longer trying to optimize the objective function".

2 pts: Answers False, clear and complete explanation.

Model 2pt answer:

Overfitting is less likely when we have a larger training set. This is because a large training set will better represent the true underlying probability distribution, thus diminishing the effect of noise uncorrelated to the target label. Since p(visit | x) is trained with more examples than p(purchase | visit, x), the former is less likely to overfit.

Part b.

0 pts: Answers False.

1 pt: Answers True, reasoning incorrect or elusive in details. e.g. "The risk of overfitting increases because the new features present in the training set may not be indicative of those on the test set, thus causing us to find spurious patterns".

2 pts: Answers True, clear and complete explanation.

Model 2pt answer:

With twice as many features, there are twice as many parameters to learn; so, for both classifiers there is a higher chance of observing a spurious correlation when the number of training examples is fixed.