Some more queries :

1. What is the relationship between the eigenvectors in the input space to the eigenvectors in the feature space ?

2. Do we always do better for any choice of Kernel function ? How can we interpret the additional eigenvalues if there is a dimension expansion ?

3. Suggestion :

Instead of using Kernel PCA for dimension expansion and for unravelling additional structure (features) in the data, can we use it for dimensionality reduction. i.e., if it is known that the input space is rank deficient, can we come up with a kernel which will map the input to a lower dimensional subspace. One obvious approach is to project the input on the subspace of eigenvectors with non zero eigen values, but this would be a linear operation. Can we come up with non-linear dimensionality reduction schemes using Kernel PCA. This Kernel in conjunction with traditional clustering schemes like k-means or EM can solve many of the instablity problems.