Edited by author 09-26-2001 11:03 PM
Hi Joe et al,

A few things I'd love to hear anyone's thoughts on:

From the perspective of "modeling" the input space, linear PCA, if I understand it right, essentially assumes the input data is a Gaussian blob centered about the origin, and the eigenvectors/values returned represent the scaling of that blob in various directions, in descending order of magnitude. In that sense, then, is the chief aim of KPCA to project a non-gaussian input distribution into gaussian distribution (typically in a higher dimensional space, but it could just as well be the same or lower depending on the mapping)? And if so, would the utility of a particular kernel be related to how well it unwraps a particular distribution into a gaussian shape, and correspondingly does that imply an inescapable tie between kernel choice and problem domain, or might there be a "universal" mapping which tends to map everything into gaussians (i.e., perhaps some of the infinite dimensional mappings have this property?)?

For a converse question, is it conceivable that certain distributions could be strongly resistant to ever appearing gaussian? One possible example is what I might call the "curse of degeneracy", which is perhaps what the curse of dimensionality becomes in KPCA: Consider an image of a tic-tac-toe board, but with no particular rules so each square either gets an X or an O. If viewed locally, each sub-image (one square) has two very obvious clusters ("X" and "O"), but the ensemble, rather than having 18 clusters, has 2^9 clusters which are annoyingly space filling in the sense of occupying all of the corners of the hypercube spanned by the 9 (in the optimal case) feature axis. (What's the plural of axis?) Worse, because of the uniformity of the independent distributions (each square choosing X with 50% odds, say), the covariance matrix would be degenerate -- and in a manner which a "generic" (i.e., with no tricks to specifically address this problem) non-linear kernel may not change since the root of the problem is that any two non-identical tic-tac-toe boards are going to have orthogonal projections onto the data set, pretty much regardless of the kernel as long as the kernel is centered and does not bias one square over another. The net result--and please correct me if I'm wrong--is that whatever eigenvectors you get will be pretty much random depending on the particulars of the dataset, but that what you won't get is 9 clear winners representing the 9 independent squares. If there's some way that KPCA does solve this, I'd love to know. It's a toy pathalogical example, but I think it has analogies in real-world images. (Perhaps that's a faulty assumption?)

Thoughts?