Regarding the issue of choosing kernels, wouldn't the kind of differential geometry based analysis done in section II-E give some idea how well a given kernel should work. As an example when I was playing around with KPCA on simple datasets (e.g. Fischer's iris data) I noticed that some kernels (e.g. homogenous polynomial and gaussian) had an apparent tendency of mapping multiple data points in the input space to a very small region in the feature space. This could possibly correspond to the kind of singularities in the feature space discussed in section II-E. Avoiding this kind of singularities would be quite useful especially if the input space data points are from different classes and one wants to use the KPCA projections as inputs to a classifier. However, it seems unlikely that there exists a simple way to use e.g. the curvature tensor R of a kernel mapping to decide what would be the best kernel for a given case.