PCA and Gaussian sources :

In my opinion, PCA does not assume any source model on the input space per se (surprise, surprise). In fact, this has been cited as one of the drawbacks of the conventional PCA by Tipping and Bishop in their paper titled Probabilistic PCA where they assume a latent variable source model.

Then why do people keep connecting PCA to Gaussian sources ? The reason is, depending on the application it is employed for, PCA has some additional special properties (such as global optimality w.r.t a cost criterion) when the source is jointly Gaussian. Let me explain this statement with a compression example.

One way to look at PCA is as a transform (called the Karhunen Leove transform) which rotates the axes of cordinates. In low complexity compression, one usually rotates the data and then compresses each dimension independently. It is a standard compression result, that when the sources are jointly Gaussian, the KL transform (PCA) is the globally optimal way to rotate the data among all transforms.

Similar results exist for other applications (such as minimum mean square estimation, classification) where PCA emerges as the optimal candidate when the sources are jointly Gaussian.