# Reproducing kernel Hilbert space

Figure illustrates related but varying approaches to viewing RKHS

In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space (RKHS) is a Hilbert space associated with a kernel that reproduces every function in the space or, equivalently, where every evaluation functional is bounded. The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman. [1]

These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated Representer theorem which states that every function in an RKHS can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem.

For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions. [2]

## Definition

Let X be an arbitrary set and H a Hilbert space of real-valued functions on X. The evaluation functional over the Hilbert space of functions H is a linear functional that evaluates each function at a point x,

${\displaystyle L_{x}:f\mapsto f(x){\text{ }}\forall f\in H.}$

We say that H is a reproducing kernel Hilbert space if ${\displaystyle L_{x}}$ is a continuous function for any x in X or, equivalently, if ${\displaystyle L_{x}}$ is a bounded operator so that for any x in X there exists some M > 0 such that

While property (Template:EquationNote) is the weakest condition that ensures both the existence of an inner product and the evaluation of every function in H at every point in the domain, it does not lend itself to easy application in practice. A more intuitive definition of the RKHS can be obtained by observing that this property guarantees that the evaluation functional can be represented by taking the inner product of ${\displaystyle f}$ with a function ${\displaystyle K_{x}}$ in H . This function is the so-called reproducing kernel for the Hilbert space H from which the RKHS takes its name. More formally, the Riesz representation theorem implies that for all x in X there exists a unique element ${\displaystyle K_{x}}$ of H with the reproducing property,

Since ${\displaystyle K_{x}}$ is itself a function in H we have that for each x in X

${\displaystyle K_{x}(y)=\langle K_{y},\ K_{x}\rangle _{H}.}$

This allows us to define the reproducing kernel of H as a function ${\displaystyle K:X\times X\to \mathbb {R} }$ by

${\displaystyle K(x,y)=K_{x}(y).}$

From this definition it is easy to see that a function ${\displaystyle K:X\times X\to \mathbb {R} }$ is a reproducing kernel if it is both symmetric and positive definite, i.e.

${\displaystyle \sum _{i,j=1}^{n}c_{i}c_{j}K(x_{i},x_{j})\geq 0}$

## Example, Counterexample

To gain intuition for the RKHS we first consider the space of square integrable functions ${\displaystyle L_{2}[a,b]}$. While a Hilbert space, ${\displaystyle L_{2}[a,b]}$ is not a RKHS as the evaluation functional is not bounded. Let ${\displaystyle c}$ be a real number in ${\displaystyle [a,b]}$, ${\displaystyle \delta }$ be the Delta function and define

${\displaystyle f(x)=x^{2}+\delta (x-c).}$

Then ${\displaystyle f(x)}$ is an element of ${\displaystyle L_{2}[a,b]}$ but there is no ${\displaystyle M}$ such that (Template:EquationNote) holds as ${\displaystyle f(c)=\infty }$. In fact this function is not even defined pointwise.

On the other hand, the space of bandlimited functions ${\displaystyle H}$ is a RKHS by shrinking the space of square integrable real-valued functions to include only those with Fourier transforms with bounded support. That is, let

${\displaystyle H=\{f\in L_{2}(\mathbb {R} )|F(\omega )\in [-a,a],a<\infty \}}$

where ${\displaystyle F(\omega )=\int f(t)e^{-i\omega t}dt}$ is the Fourier transform of ${\displaystyle f}$. One can show that if ${\displaystyle f\in H}$ then

${\displaystyle f(x)={\frac {1}{2\pi }}\int _{-a}^{a}\phi (\omega )e^{ix\omega }d\omega }$

for ${\displaystyle \phi \in L_{2}[-a,a]}$. It then follows that

${\displaystyle |f(x)|\leq {\sqrt {\frac {1}{2\pi }}}{\sqrt {{\frac {1}{2\pi }}\int _{-a}^{a}|\phi (\omega )|^{2}d\omega }}={\sqrt {\frac {1}{2\pi }}}\|f\|.}$

As this inequality shows that the evaluation functional is bounded and ${\displaystyle H}$ is also a Hilbert space, ${\displaystyle H}$ is indeed a RKHS.

## Moore-Aronszajn Theorem

We have seen how a reproducing kernel Hilbert space defines a reproducing kernel function that is both symmetric and positive definite. The Moore-Aronszajn theorem goes in the other direction; it states that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.

Theorem. Suppose K is a symmetric, positive definite kernel on a set X. Then there is a unique Hilbert space of functions on X for which K is a reproducing kernel.

Proof. For all x in X, define Kx = K(x, ⋅ ). Let H0 be the linear span of {Kx : xX}. Define an inner product on H0 by

${\displaystyle \left\langle \sum _{j=1}^{n}b_{j}K_{y_{j}},\sum _{i=1}^{m}a_{i}K_{x_{i}}\right\rangle =\sum _{i=1}^{m}\sum _{j=1}^{n}{a_{i}}b_{j}K(y_{j},x_{i}).}$

The symmetry of this inner product follows from the symmetry of K and the non-degeneracy follows from the fact that K is positive definite.

Let H be the completion of H0 with respect to this inner product. Then H consists of functions of the form

${\displaystyle f(x)=\sum _{i=1}^{\infty }a_{i}K_{x_{i}}(x)}$

where ${\displaystyle \sum _{i=1}^{\infty }a_{i}^{2}K(x_{i},x_{i})<\infty }$. The fact that the above sum converges for every x follows from the Cauchy-Schwarz inequality.

Now we can check the reproducing property (Template:EquationNote):

${\displaystyle \langle f,K_{x}\rangle =\left\langle \sum _{i=1}^{\infty }a_{i}K_{x_{i}},K_{x}\right\rangle =\sum _{i=1}^{\infty }a_{i}K(x_{i},x)=f(x).}$

To prove uniqueness, let G be another Hilbert space of functions for which K is a reproducing kernel. For any x and y in X, (Template:EquationNote) implies that

${\displaystyle \langle K_{x},K_{y}\rangle _{H}=K(x,y)=\langle K_{x},K_{y}\rangle _{G}.\,}$

By linearity, ${\displaystyle \langle \cdot ,\cdot \rangle _{H}=\langle \cdot ,\cdot \rangle _{G}}$ on the span of {Kx : xX}. Then G = H by the uniqueness of the completion.

## Integral Operators and Mercer's Theorem

We may characterize a symmetric positive definite kernel ${\displaystyle K}$ via the integral operator using Mercer's theorem and obtain an additional view of the RKHS. Let ${\displaystyle X}$ be a compact space equipped with a strictly positive finite Borel measure ${\displaystyle \mu }$ and ${\displaystyle K:X\times X\to \mathbb {R} }$ a continuous, symmetric, and positive definite function. Define the integral operator ${\displaystyle T_{K}:L_{2}(X)\rightarrow L_{2}(X)}$ as

${\displaystyle [T_{K}f](\cdot )=\int _{X}K(\cdot ,t)f(t)\,d\mu (t)}$

where ${\displaystyle L_{2}(X)}$ is the space of square integrable functions with respect to ${\displaystyle \mu }$.

Mercer's theorem states that the spectral decomposition of the integral operator ${\displaystyle T_{K}}$ of ${\displaystyle K}$ yields a series representation of ${\displaystyle K}$ in terms of the eigenvalues and eigenfunctions of ${\displaystyle T_{K}}$. This then implies that ${\displaystyle K}$ is a reproducing kernel so that the corresponding RKHS can be defined in terms of these eigenvalues and eigenfunctions. We provide the details below.

Under these assumptions ${\displaystyle T_{k}}$ is a compact, continuous, self-adjoint, and positive operator. The spectral theorem for self-adjoint operators implies that there is an at most countable decreasing sequence ${\displaystyle (\sigma _{i})_{i}\geq 0}$ such that ${\displaystyle \lim _{i\to \infty }\sigma _{i}=0}$ and ${\displaystyle T_{K}\phi _{i}(x)=\sigma _{i}\phi _{i}(x)}$, where the ${\displaystyle \{\phi _{i}\}}$ form an orthonormal basis of ${\displaystyle L_{2}(X)}$. By the positivity ${\displaystyle T_{k}}$, ${\displaystyle \sigma _{i}>0{\text{ }}\forall i}$. One can also show that ${\displaystyle T_{k}}$ maps continuously into the space of continuous functions ${\displaystyle C(X)}$ and therefore we may choose continuous functions as the eigenvectors, that is, ${\displaystyle \phi _{i}\in C(X){\text{ }}\forall i}$. Then by Mercer's theorem ${\displaystyle K}$ may be written in terms of the eigenvalues and continuous eigenfunctions as

${\displaystyle K(x,y)=\sum _{j=1}^{\infty }\sigma _{j}\,\phi _{j}(x)\,\phi _{j}(y)}$

for all ${\displaystyle x,y}$ in ${\displaystyle X}$ such that ${\displaystyle \lim _{n\to \infty }\sup _{u,v}|K(u,v)-\sum _{j=1}^{n}\sigma _{j}\,\phi _{j}(u)\,\phi _{j}(v)|=0.}$ This above series representation is referred to as a Mercer kernel or Mercer representation of ${\displaystyle K}$.

Furthermore, it can be shown that the RKHS ${\displaystyle H}$ of ${\displaystyle K}$ is given by

${\displaystyle H=\left\{f{\in }L_{2}(X){\mathrel {\Bigg |}}\sum _{i=1}^{\infty }{\frac {\left\langle f,\phi _{i}\right\rangle ^{2}}{\sigma _{i}}}<\infty \right\}}$

where the inner product of ${\displaystyle H}$ given by ${\displaystyle \left\langle f,g\right\rangle _{H}=\sum _{i=1}^{\infty }{\frac {\left\langle f,\phi _{i}\right\rangle _{L_{2}}\left\langle g,\phi _{i}\right\rangle _{L_{2}}}{\sigma _{i}}}.}$ This representation of the RKHS has application in probability and statistics, for example to the Karhunen-Loeve representation for stochastic processes and kernel PCA.

## Feature Maps

A feature map is a map ${\displaystyle \varphi :X\rightarrow F}$, where ${\displaystyle F}$ is a Hilbert space which we will call the feature space. The first sections presented the connection between bounded/continuous evaluation functions, positive definite functions, and integral operators and in this section we provide another representation of the RKHS in terms of feature maps.

We first note that every feature map defines a kernel via

Clearly ${\displaystyle K}$ is symmetric and positive definiteness follows from the properties of inner product in ${\displaystyle F}$. Conversely, every positive definite function and corresponding reproducing kernel Hilbert space has infinitely many associated feature maps such that (Template:EquationNote) holds.

For example, we can trivially take ${\displaystyle F=H}$ and ${\displaystyle \varphi (x)=K_{x}}$ for all ${\displaystyle x\in X}$. Then (Template:EquationNote) is satisfied by the reproducing property. Another classical example of a feature map relates to the previous section regarding integral operators by taking ${\displaystyle F=\ell ^{2}}$ and ${\displaystyle \varphi (x)({\sqrt {(}}\sigma _{i})\phi _{i}(x))_{i}}$.

This connection between kernels and feature maps provides us with a new way to understand positive definite functions and hence reproducing kernels as inner products in ${\displaystyle H}$. Moreover, every feature map can naturally define a RKHS by means of the definition of a positive definite function.

Lastly, feature maps allow us to construct function spaces that reveal another perspective on the RKHS. Consider the linear space

${\displaystyle H_{\varphi }=\{f:X\to \mathbb {R} |\exists w\in F,f(x)=_{F},\forall {\text{ }}x\in X\}.}$

We can define a norm on ${\displaystyle H_{\varphi }}$ by

${\displaystyle ||f||_{\varphi }={\text{inf}}\{||w||_{F}:w\in F,f(x)=_{F},\forall {\text{ }}x\in X\}.}$

It can be shown that ${\displaystyle H_{\varphi }}$ is a RKHS with kernel defined by ${\displaystyle K(x,y)=<\varphi (x),\varphi (y)>}$. This representation implies that the elements of the RKHS are inner products of elements in the feature space and can accordingly be seen as hyperplanes. This view of the RKHS is related to the kernel trick in machine learning. [4]

## Properties

The following properties of RKHSs may be useful to readers.

${\displaystyle K((x_{1},\dots ,x_{p}),(y_{1},\dots ,y_{p}))=K_{1}(x_{1},y_{1})\dots K_{p}(x_{p},y_{p})}$
${\displaystyle d_{K}(x,y)=||K_{x}-K_{y}||_{H}^{2}=2(1-K(x,y)){\text{ }}\forall x\in X}$.

By the Cauchy-Schwartz inequality,

${\displaystyle K(x,y)^{2}\leq K(x,x)K(y,y){\text{ }}\forall x,y\in X.}$

This inequality allows us to view ${\displaystyle K}$ as a measure of similarity between inputs. If ${\displaystyle x,y\in X}$ are similar then ${\displaystyle K(x,y)}$ will be closer to 1 while if ${\displaystyle x,y\in X}$ are dissimilar then ${\displaystyle K(x,y)}$ will be closer to 0.

## Examples

Common examples of kernels include:

• Linear Kernel:
${\displaystyle K(x,y)=\langle x,y\rangle }$
• Polynomial Kernel:
${\displaystyle K(x,y)=(\alpha \langle x,y\rangle +1)^{d},\alpha {\in }\mathbb {R} ,d{\in }\mathbb {N} }$

Other common examples are kernels which satisfy ${\displaystyle K(x,y)=K(\|x-y\|)}$. These are the radial basis function kernels.

• Gaussian Kernel:
Sometimes referred to as the Radial basis function kernel, or squared exponential kernel
${\displaystyle K(x,y)=e^{-{\frac {\|x-y\|^{2}}{2\sigma ^{2}}}},\sigma >0}$
• Laplacian Kernel:
${\displaystyle K(x,y)=e^{-{\frac {\|x-y\|}{\sigma }}},\sigma >0}$

We also provide examples of Bergman kernels. Let X be finite and let H consist of all complex-valued functions on X. Then an element of H can be represented as an array of complex numbers. If the usual inner product is used, then Kx is the function whose value is 1 at x and 0 everywhere else, and K(x,y) can be thought of as an identity matrix since K(x,y)=1 when x=y and K(x,y)=0 otherwise. In this case, H is isomorphic to Cn.

The case of X = D is more sophisticated, here the Bergman space H2(D) is the space of square-integrable holomorphic functions on D. It can be shown that the reproducing kernel for H2(D) is

${\displaystyle K(x,y)={\frac {1}{\pi }}{\frac {1}{(1-x{\overline {y}})^{2}}}.}$

Lastly, the space of band limited functions ${\displaystyle f}$ in ${\displaystyle L^{2}(\mathbb {R} )}$ with bandwidth ${\displaystyle \pi }$ are a RKHS with reproducing kernel

${\displaystyle K(x,y)={\frac {\sin \pi (x-y)}{\pi (x-y)}}.}$

## Extension to Vector-Valued Functions

In this section we extend the definition of the RKHS to spaces of vector-valued functions as this extension is particularly important in multi-task learning and manifold regularization. The main difference is that the reproducing kernel ${\displaystyle \Gamma }$ is a symmetric function that is now a positive semi-definite matrix for any ${\displaystyle x,y}$ in ${\displaystyle X}$. More formally, we define a vector-valued RKHS (vvRKHS) as a Hilbert space of functions ${\displaystyle f:X\to \mathbb {R} ^{T}}$ such that for all ${\displaystyle c\in \mathbb {R} ^{T}}$ and ${\displaystyle x\in X}$

${\displaystyle \Gamma _{x}c(y)=\Gamma (x,y)c\in H{\text{ for }}y\in X}$

and

${\displaystyle _{H}=f(x)^{\intercal }c.}$

This second property parallels the reproducing property for the scalar-valued case. We note that this definition can also be connected to integral operators, bounded evaluation functions, and feature maps as we saw for the scalar-valued RKHS. We can equivalently define the vvRKHS as a vector-valued Hilbert space with a bounded evaluation functional and show that this implies the existence of a unique reproducing kernel by the Riesz Representation theorem. Mercer's theorem can also be extended to address the vector-valued setting and we can therefore obtain a feature map view of the vvRKHS. Lastly, it can also be shown that the closure of the span of ${\displaystyle \{\Gamma _{x}c:x\in X,c\in \mathbb {R} ^{T}\}}$ coincides with ${\displaystyle H}$, another property similar to the scalar-valued case.

We can gain intuition for the vvRKHS by taking a component-wise perspective on these spaces. In particular, we find that every vvRKHS is isometrically isomorphic to a scalar-valued RKHS on a particular input space. Let ${\displaystyle \Lambda =\{1,\dots ,T\}}$. Consider the space ${\displaystyle X\times \Lambda }$ and the corresponding reproducing kernel

As noted above, the RKHS associated to this reproducing kernel is given by the closure of the span of ${\displaystyle \{\gamma _{(x,t)}:x\in X,t\in \Lambda \}}$ where ${\displaystyle \ \gamma _{(x,t)}(y,s)=\gamma ((x,t),(y,s))}$ for every set of pairs ${\displaystyle (x,t),(y,s)\in X\times \Lambda }$.

The connection to the scalar-valued RKHS can then be made by the fact that every matrix-valued kernel can be identified with a kernel of the form of (Template:EquationNote) via

${\displaystyle \Gamma (x,y)_{(t,s)}=\gamma ((x,t),(y,s)).}$

Moreover, every kernel with the form of (Template:EquationNote) defines a matrix-valued kernel with the above expression. Now letting the map ${\displaystyle D:H_{\Gamma }\to H_{\gamma }}$ be defined as

${\displaystyle (Df)(x,t)=_{\mathbb {R} ^{T}}}$

where ${\displaystyle e_{t}}$ is the ${\displaystyle t^{th}}$ component of the canonical basis for ${\displaystyle \mathbb {R} ^{T}}$, one can show that ${\displaystyle D}$ is bijective and an isometry between ${\displaystyle H_{\Gamma }}$ and ${\displaystyle H_{\gamma }}$.

While this view of the vvRKHS can be quite useful in multi-task learning, it should be noted that this isometry does not reduce the study of the vector-valued case to that of the scalar-valued case. In fact, this isometry procedure can make both the scalar-valued kernel and the input space too difficult to work with in practice as properties of the original kernels are often lost. [6] [7] [8]

An important class of matrix-valued reproducing kernels are separable kernels which can factorized as the product of a scalar valued kernel and a ${\displaystyle T}$-dimensional symmetric positive semi-definite matrix. In light of our previous discussion these kernels are of the form

${\displaystyle \gamma ((x,t),(y,s))=K(x,y)K_{T}(t,s)}$

for all ${\displaystyle x,y}$ in ${\displaystyle X}$ and ${\displaystyle t,s}$ in ${\displaystyle T}$. As the scalar-valued kernel encodes dependencies between the inputs, we can observe that the matrix-valued kernel encodes dependencies among both the inputs and the outputs.

We lastly remark that the above theory can be further extended to spaces of functions with values in function spaces but obtaining kernels for these spaces is a more difficult task. [9]

1. Okutmustur
2. Paulson
3. Durrett
4. Rosasco
5. Rosasco
6. De Vito
7. Zhang
8. Alvarez
9. Rosasco

## References

• Alvarez, Mauricio, Rosasco, Lorenzo and Lawrence, Neil, “Kernels for Vector-Valued Functions: a Review,” http://arxiv.org/abs/1106.6251, June 2011.
• {{#invoke:Citation/CS1|citation

|CitationClass=journal }}

• Berlinet, Alain and Thomas, Christine. Reproducing kernel Hilbert spaces in Probability and Statistics, Kluwer Academic Publishers, 2004.
• {{#invoke:Citation/CS1|citation

|CitationClass=journal }}

|CitationClass=journal }}

|CitationClass=journal }}

• Rosasco, Lorenzo and Poggio, Thomas. "A Regularization Tour of Machine Learning - MIT 9.520 Lecture Notes" Manuscript, Dec. 2014.
• Wahba, Grace, Spline Models for Observational Data, SIAM, 1990.
• Zhang, Haizhang, Xu, Yuesheng, and Zhang, Qinghui (2012). "Refinement of Operator-valued Reproducing Kernels." Journal of Machine Learning Research 13 91-136.