I have two comments about the example in page 9.

1. A very simple question: I hope we can spend sometime in class to learn to inteprete the result of Kernel PCA, because I find it harder to understand
than linear PCA. The caption of Figure 2 says "Non-linear PCA uses the third component to pick up the variance caused by the noise, as can be seen in the case of degree 2." Hmm, I don't think I can see that from the figure. Does it mean that pertubations caused by the noise tend to have similar projections on the third eigenvector? If that's the case then this noise is not well-captured by the d=3 and d=4 case.

2. I found it very interesting that the contour lines are simply hyperbolics and concentric circles. I understand that the contour lines depend on a.) the distribution of the data, which is artificial in this example, and b.) the kernel. But somehow my intuition is that these kind of contours are quite common in the polynomia kernel. Is there any heuristic that would predict the, say, concentric contours in the second component in the d=2 case?

An additional suggestion is: has anyone tried to apply kernel PCA to natural images, and see what the components look like (a la Bell and Sejnowski in their ICA paper)? We can try polynomial kernel with different order (d=2,3,4,5...) and see which develops simple-cell-like receptive fields. This way we can infer that maybe only statistical relationship of a certain order is sufficient to account for the emergence of edge detectors.

Also, single-unit recording showed that neurons in visual area V4 are activated by concentric circles and hyperbolas. I did a project to show that these higer order features can indeed help object recognition. Maybe this can be explained in terms of kernel PCA (eg. detecting statistical relationship of a certain order)?