/m11 answer: The Laplace distribution has finite variance, and tails that are heavier than the Gaussian's.

The Pareto distribution has even heavier tails. See pages 623 and 625 of Casella and Berger.

In ridge regression, I do not see the point of standardizing the data by shifting and scaling. If we shift the data, the b values do not change, except b0. If we scale it by 1/std of the column then b values are scaled by std of column itself? But we still have the same predicted y vector. So, what do we gain by standardizing?

For problem 2 part b, is using the absolute value of the difference between two medians as our test statistic good idea? Or should it be something more complicated?

Banu

For Problem 2a)

What does it mean to estimate the median of a distribution using bootstrapping. I can see how you can estimate the median from a sample. I don't see how bootstraping helps. Bootstrapping may be useful to figure out the distribution of the median. Is that the question?

-Jan

/m16 The scaling is important to give each feature an equal chance at contributing.

Let's say feature 1 is an excellent predictor of the output, however, the variance of feature 1 is tiny. Then in order to fit a good model, the b coefficient of this feature will have to be huge. However, in ridge regression we are trying to also shrink b as we fit the model, so a ridge fitted model may ignore feature 1 in favor of other features which have a bigger variance even though they are worse predictors. Normalizing each feature eliminates this problem by making the 'average' b for each feature about the same.

-Jan

Re: LaPlace

The LaPlace (double-exponential) distribution seems to have tails that are skinnier than the Gaussian. Is there a way to shape it such that it has heavy tails (ideally with variance=1)?

/m20 answer: The Laplace has fatter tails than the Gaussian because the probability of x decays as exp(-x) for the Laplace, as opposed to exp(-x^2) for the Gaussian.

/m16, /m19: Thanks for the explanation, Jan.

/m17 answer: Using the difference between the two medians as your test statistic is reasonable, but only if the distributions have similar spread.

Otherwise, you may want to be inspired by a known test for whether two medians are equal, e.g. the Wilcoxon rank sum test. See http://www.mathworks.com/access/helpdesk/h.../stats/ranksum.html

/m18 answer: You are right, I mean use the bootstrap method to investigate the distribution of the sample median.

Edited by author 03-15-2005 08:13 PM
/m23 As far as I know, Wilcoxon rank-sum test is a nonparametric test to see if two distributions are independent or not (like t-test). I think it's quite different from testing if two medians are the same. For example, if a = [0 0 0 0 0 1 2 3 4], b = [-4 -3 -2 -1 0 0 0 0 0], then rank-sum test would report some 'significant' result, but actually their median is the same. Shouldn't we use a different test?

About Message 18.
If you want us to estimate the distribution of the median in Pb 2, do you then just want us to this for a few different sample sizes. (You said a range of sample sizes, which seem to mean that you would like to see the behavior over different sizes, but if we have to estimate the actual distribution for each sample size, it'll be hard to report as a function over a range...)
An alternative would be to estimate the confidence intervals obtained for the median, and report this over a range of sample sizes. Would this be fine?

About Pb 1,
what do you mean by "error bars" for giving a forecast of the MSE of our final model ?

/m25 reply: I think Hyun Min Kang is right that the Wilcoxon rank-sum test is not a test for whether or not two medians are the same in general. It is a test of H0: two distributions have the same shape and the same median versus H1: the two distributions have the same shape but different medians.

/m26 answer: For 2(a) we are interested in the variance of the bootstrap-based estimate of the median, compared to the variance of (for example) the sample mean. Each of these variances is a function of the sample size N, so the efficiency is a function of N.

For 1, I mean give a forecast confidence interval of the MSE of your model, so you can evaluate if the true MSE falls within this confidence interval.