Mercer theorem measure and RKHS norm

2023/04/12

This problem from the GPML book relates to the effect of the choice of measure when using Mercer’s theorem to compute kernel eigenfunctions on the resulting norm in the RKHS induced by that kernel. In the problem we show that in the finite dimensional case (this also applies in the ∞ \infty dimensions case but it is then harder to show), the RKHS norm is independent of the measure chosen. This is interesting to me because when first learning about Mercer’s theorem, I was confused by how the measure the eigenfunctions are computed w.r.t is chosen when using the theorem in practice. This question shows that in for some important applications, the measure doesn’t matter.

Problem

We motivate the fact that the RKHS norm does not depend on the density p(x) p(x) using a finite-dimensional analogue. Consider the n-dimensional vector f \mathbf f , and let the n×n n \times n matrix Φ \Phi be comprised of non-colinear columns ϕ1,…,ϕn \phi_1,\dots, \phi_n . Then f \mathbf f can be expressed as a linear combination of these basis vectors f=∑i=1nciϕi=Φc \mathbf f = \sum^n_{i=1} c_i\phi_i = \Phi \mathbf c for some coefficients ci \\{c_i\\} . Let the ϕ \phi s be eigenvectors of the covariance matrix K K w.r.t. a diagonal matrix P P with non-negative entries, so that KPΦ=ΦΛ KP \Phi = \Phi\Lambda , where Λ \Lambda is a diagonal matrix containing the eigenvalues. Note that Φ⊤PΦ=I \Phi^\top P\Phi = I . Show that ∑i=1nci2/λi=c⊤Λ−1c=f⊤K−1f \sum^n_{i=1}c^2_i/\lambda_i=\mathbf c^\top\Lambda^{-1}\mathbf c=\mathbf f^\top K^{-1} \mathbf f , and thus observe that f⊤K−1f \mathbf f^\top K^{-1} \mathbf f can be expressed as c⊤Λc \mathbf c^\top\Lambda\mathbf c for any valid P P and corresponding Φ \Phi . (Note the matrix P P here represents the measure.)

Source: Gaussian Processes for Machine Learning by Rasmussen and Williams, Ex. 6.7.1

Solution

This solution is a bit strange as I got stuck and this is the only way I could think to do it, I am sure there is a better way. First using that Φ⊤PΦ=I \Phi^\top P\Phi = I we can see that,

KPΦ=ΦΛ  ⟹  Φ⊤PKPΦ=Λ. KP\Phi=\Phi\Lambda \implies \Phi^\top PKP\Phi=\Lambda.

We can then write

c⊤Λ−1c=c⊤(Φ⊤PKPΦ)−1c=c⊤Φ−1P−1K−1P−1Φ−⊤c=c⊤Φ⊤PP−1K−1P−1PΦc=f⊤K−1f \begin{align} \mathbf c^\top \Lambda^{-1} \mathbf c &= \mathbf c^\top (\Phi^\top PKP\Phi)^{-1} \mathbf c \\ &= \mathbf c^\top \Phi^{-1}P^{-1}K^{-1}P^{-1}\Phi^{-\top} \mathbf c \\ &= \mathbf c^\top \Phi^\top P P^{-1}K^{-1}P^{-1} P \Phi \mathbf c \\ &=\mathbf f^{\top} K^{-1} \mathbf f \end{align}

as required, where on the second to last line we have twice used the fact that Φ−1=Φ⊤P \Phi^{-1}=\Phi^\top P .