# Real reproducing kernel Hilbert spaces

Jordan Bell
October 22, 2015

## 1 Reproducing kernels

We shall often speak about functions $F:X\times X\to\mathbb{R}$, where $X$ is a nonempty set. For $x\in X$, we define $F_{x}:X\to\mathbb{R}$ by $F_{x}(y)=F(x,y)$ and for $y\in X$ we define $F^{y}:X\to\mathbb{R}$ by $F^{y}(x)=F(x,y)$. $F$ is said to be symmetric if $F(x,y)=F(y,x)$ for all $x,y\in X$ and positive-definite if for any $x_{1},\ldots,x_{n}\in X$ and $c_{1},\ldots,c_{n}\in\mathbb{R}$ it holds that

 $\sum_{1\leq i,j\leq n}c_{i}c_{j}F(x_{i},x_{j})\geq 0.$
###### Lemma 1.

If $F:X\times X\to\mathbb{R}$ is symmetric and positive-definite then

 $F(x,y)^{2}\leq F(x,x)F(y,y),\qquad x,y\in X.$
###### Proof.

For $\alpha,\beta\in\mathbb{R}$ define11 1 See Alain Berlinet and Christine Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics, p. 13, Lemma 3.

 $\displaystyle C(\alpha,\beta)$ $\displaystyle=\alpha^{2}F(x,x)+\alpha\beta F(x,y)+\beta\alpha F(y,x)+\beta^{2}% F(y,y)$ $\displaystyle=\alpha^{2}F(x,x)+2\alpha\beta F(x,y)+\beta^{2}F(y,y),$

which is $\geq 0$. Let

 $\displaystyle P(\alpha)$ $\displaystyle=C(\alpha,F(x,y))$ $\displaystyle=\alpha^{2}F(x,x)+2\alpha F(x,y)^{2}+F(x,y)^{2}F(y,y),$

which is $\geq 0$. In the case $F(x,x)=0$, the fact that $P\geq 0$ implies that $F(x,y)=0$. In the case $F(x,y)\neq 0$, $P(\alpha)$ is a quadratic polynomial and because $P\geq 0$ it follows that the discriminant of $P$ is $\leq 0$:

 $4F(x,y)^{4}-4\cdot F(x,x)\cdot F(x,y)^{2}F(y,y)\leq 0.$

That is, $F(x,y)^{4}\leq F(x,y)^{2}F(x,x)F(y,y)$, and this implies that $F(x,y)^{2}\leq F(x,x)F(y,y)$. ∎

A real reproducing kernel Hilbert space is a Hilbert space $H$ contained in $\mathbb{R}^{X}$, where $X$ is a nonempty set, such that for each $x\in X$ the map $\Lambda_{x}f=f(x)$ is continuous $H\to\mathbb{R}$. In this note we speak always about real Hilbert spaces.

Let $H\subset\mathbb{R}^{X}$ be a reproducing kernel Hilbert space. Because $H$ is a Hilbert space, the Riesz representation theorem states that $\Phi:H\to H^{*}$ defined by

 $(\Phi g)(f)=\left\langle f,g\right\rangle_{H},\qquad g,f\in H$

is an isometric isomorphism. Because $H$ is a reproducing kernel Hilbert space, $\Lambda_{x}\in H^{*}$ for each $x\in X$ and we define $T_{x}=\Phi^{-1}\Lambda_{x}\in H$, which satisfies

 $f(x)=\Lambda_{x}(f)=\left\langle f,T_{x}\right\rangle_{H},\qquad f\in H.$

In particular, because $T_{x}\in H$, for $y\in X$ it holds that

 $T_{x}(y)=\Lambda_{y}(T_{x})=\left\langle T_{x},T_{y}\right\rangle_{H}.$

Define $K:X\times X\to\mathbb{R}$ by

 $K(x,y)=\left\langle T_{x},T_{y}\right\rangle_{H},$

called the reproducing kernel of $H$. For $x,y\in X$,

 $T_{x}(y)=\left\langle T_{x},T_{y}\right\rangle_{H}=K(x,y)=K_{x}(y),$

which means that $T_{x}=K_{x}$.

A reproducing kernel is symmetric and positive-definite:

 $K(x,y)=\left\langle T_{x},T_{y}\right\rangle_{H}=\left\langle T_{y},T_{x}% \right\rangle_{H}=K(y,x)$

and

 $\displaystyle\sum_{1\leq i,j\leq n}c_{i}c_{j}K(x_{i},x_{j})$ $\displaystyle=\sum_{1\leq i,j\leq n}\left\langle c_{i}T_{x_{i}},c_{j}T_{x_{j}}% \right\rangle_{H}$ $\displaystyle=\left\langle\sum_{1\leq i\leq n}c_{i}T_{x_{i}},\sum_{1\leq j\leq n% }c_{j}T_{x_{j}}\right\rangle_{H}$ $\displaystyle\geq 0.$
###### Lemma 2.

If $E$ is an orthonormal basis for a reproducing kernel Hilbert space $H\subset\mathbb{R}^{X}$ with reproducing kernel $K:X\times X\to\mathbb{R}$, then

 $K(x,y)=\sum_{e\in E}e(x)e(y),\qquad x,y\in X.$
###### Proof.

Because $E$ is an orthonormal basis for $H$, Parseval’s identity tell us

 $\left\langle T_{x},T_{y}\right\rangle_{H}=\sum_{e\in E}\left\langle T_{x},e% \right\rangle\left\langle T_{y},e\right\rangle=\sum_{e\in E}\left\langle e,T_{% x}\right\rangle\left\langle e,T_{y}\right\rangle=\sum_{e\in E}e(x)e(y).$

If $H\subset\mathbb{R}^{X}$ is a reproducing kernel Hilbert space with reproducing kernel $K:X\times X\to\mathbb{R}$ and $V$ is a closed linear subspace of $H$, then $V$ is itself a reproducing kernel Hilbert space, with some reproducing kernel $G:X\times X\to\mathbb{R}$. The following theorem expresses $G$ in terms of $K$.22 2 Ward Cheney and Will Light, A Course in Approximation Theory, p. 234, Chapter 31, Theorem 4.

###### Theorem 3.

Let $H\subset\mathbb{R}^{X}$ be a reproducing kernel Hilbert space with reproducing kernel $K:X\times X\to\mathbb{R}$, let $V$ be a closed linear subspace of $H$ with reproducing kernel $G:X\times X\to\mathbb{R}$, and let $P_{V}:H\to V$ be the projection onto $V$. Then

 $G_{x}=P_{V}K_{x},\qquad x\in X.$
###### Proof.

$H=V\oplus V^{\perp}$, thus for $f\in H$ there are unique $g\in V,h\in V^{\perp}$ such that $f=g+h$, and $P_{V}f=g$.3 Then $f-P_{V}f\in V^{\perp}$. Therefore for $y\in X$, as $G_{y}\in V$ it holds that

 $\left\langle f,G_{y}\right\rangle_{H}=\left\langle f-P_{V}f+P_{V}f,G_{y}\right% \rangle_{H}=\left\langle P_{V}f,G_{y}\right\rangle_{H}=(P_{V}f)(y).$

In particular, for $x,y\in X$ and $f=K_{x}$,

 $(P_{V}K_{x})(y)=\left\langle K_{x},G_{y}\right\rangle_{H}=\left\langle G_{y},T% _{x}\right\rangle_{H}=G_{y}(x)=G(y,x)=G(x,y)=G_{x}(y),$

which means that $P_{V}K_{x}=G_{x}$, proving the claim. ∎

The Moore-Aronszajn theorem states that if $X$ is a nonempty set and $K:X\times X\to\mathbb{R}$ is a symmetric and positive-definite function, then there is a unique reproducing kernel Hilbert space $H\subset\mathbb{R}^{X}$ for which $K$ is the reproducing kernel.

We now prove that given a symmetric positive-definite kernel there is a unique reproducing Hilbert space for which it is the reproducing kernel.44 4 Alain Berlinet and Christine Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics, p. 19, Theorem 3.

## 2 Sobolev spaces on $[0,T]$

Let $f\in\mathbb{R}^{[0,T]}$. The following are equivalent:55 5 Elias M. Stein and Rami Shakarchi, Real Analysis, p. 130, Theorem 3.11.

1. 1.

$f$ is absolutely continuous.

2. 2.

$f$ is differentiable at almost all $t\in[0,T]$, $f^{\prime}\in L^{1}$, and

 $f(t)=f(0)+\int_{0}^{t}f^{\prime}(s)ds,\qquad t\in[0,T].$
3. 3.

There is some $g\in L^{1}$ such that

 $f(t)=f(0)+\int_{0}^{t}g(s)ds,\qquad t\in[0,T].$

In particular, if $f$ is absolutely continuous and $f^{\prime}=0$ almost everywhere then $\int_{0}^{t}f^{\prime}(s)ds=0$ and so $f(t)=f(0)$ for all $t\in[0,T]$. That is, if $f$ is absolutely continuous and $f^{\prime}=0$ almost everywhere then $f$ is constant.

Let $H$ be the set of those absolutely continuous functions $f\in\mathbb{R}^{[0,T]}$ such that $f(0)=0$ and $f^{\prime}\in L^{2}$. For $f,g\in H$ define

 $\left\langle f,g\right\rangle_{H}=\int_{0}^{T}f^{\prime}(s)g^{\prime}(s)ds.$

If $\left\|f\right\|_{H}=0$ then $\int_{0}^{T}f^{\prime}(s)^{2}ds=0$, which implies that $f^{\prime}=0$ almost everywhere and hence that $f$ is constant, and therefore $f=0$. Thus $\left\langle\cdot,\cdot\right\rangle_{H}$ is indeed an inner product on $H$.

If $f_{n}$ is a Cauchy sequence in $H$ then $f_{n}^{\prime}$ is a Cauchy sequence in $L^{2}$ and hence converges to some $g\in L^{2}$. Then the function $f\in\mathbb{R}^{[0,T]}$ defined by

 $f(t)=\int_{0}^{t}g(s)ds,\qquad t\in[0,T],$

is absolutely continuous, $f(0)=0$, and satisfies $f^{\prime}=g$ almost everywhere, which shows that $f\in H$. Then $f_{n}\to f$ in $H$, which proves that $H$ is a Hilbert space. For $t\in[0,T]$, by the Cauchy-Schwarz inequality,

 $|f(t)|^{2}=\left|\int_{0}^{t}f^{\prime}(s)ds\right|^{2}\leq\left|\int_{0}^{T}f% ^{\prime}(s)ds\right|^{2}\leq T\int_{0}^{T}f^{\prime}(s)^{2}ds=T\left\|f\right% \|_{H}^{2},$

i.e. $|L_{t}f|\leq T^{1/2}\left\|f\right\|_{H}$, which shows that $L_{t}\in H^{*}$. Therefore $H$ is a reproducing kernel Hilbert space.

For $a\in[0,T]$ define $h_{a}:[0,T]\to\mathbb{R}$ by $h_{a}(s)=1_{[0,a]}(s)$, which belongs to $L^{2}$, and define $g_{a}:[0,T]\to\mathbb{R}$ by

 $g_{a}(t)=\int_{0}^{t}h_{a}(s)ds=\min(t,a),$

which belongs to $H$. For $f\in H$,

 $\left\langle f,g_{a}\right\rangle_{H}=\int_{0}^{T}f^{\prime}(s)g_{a}^{\prime}(% s)ds=\int_{0}^{T}f^{\prime}(s)1_{[0,a]}(s)ds=\int_{0}^{a}f^{\prime}(s)ds=f(a).$

This means that $K_{a}=g_{a}$. For $a,b\in[0,T]$,

 $\left\langle K_{a},K_{b}\right\rangle_{H}=\int_{0}^{T}g_{a}^{\prime}(s)g_{b}^{% \prime}(s)ds=\int_{0}^{T}1_{[0,a]}(s)1_{[0,b]}(s)ds=\int_{0}^{T}1_{[0,\min(a,b% )]}(s)ds.$

That is, the reproducing kernel of $H$ is $K:[0,T]\times[0,T]\to\mathbb{R}$,

 $K(a,b)=\left\langle K_{a},K_{b}\right\rangle_{H}=\min(a,b).$

## 3 Sobolev spaces on $\mathbb{R}$

Let $\lambda$ be Lebesgue measure on $\mathbb{R}$. Let $\mathscr{L}^{2}(\lambda)$ be the collection of Borel measurable functions $f:\mathbb{R}\to\mathbb{R}$ such that $|f|^{2}$ is integrable, and let $L^{2}(\lambda)$ be the Hilbert space of equivalence classes of elements of $\mathscr{L}^{2}(\lambda)$ where $f\sim g$ when $f=g$ almost everywhere, with

 $\left\langle f,g\right\rangle_{L^{2}}=\int_{\mathbb{R}}fgd\lambda.$

Let $H^{1}(\mathbb{R})$ be the set of locally absolutely continuous functions $f:\mathbb{R}\to\mathbb{R}$ such that $f,f^{\prime}\in L^{2}(\lambda)$. This is a Hilbert space with the inner product6

 $\left\langle f,g\right\rangle_{H^{1}}=\left\langle f,g\right\rangle_{L^{2}}+% \left\langle f^{\prime},g^{\prime}\right\rangle_{L^{2}}.$

Define $K:\mathbb{R}\times\mathbb{R}\to\mathbb{R}$ by

 $K(x,y)=\frac{1}{2}\exp(-|x-y|),\qquad x,y\in\mathbb{R}.$

Let $x\in\mathbb{R}$. For $y, $K_{x}^{\prime}(y)=K_{x}(y)$ and for $y>x$, $K_{x}^{\prime}(y)=-K_{x}(y)$, which shows that $K_{x}\in H^{1}(\mathbb{R})$. For $f\in H^{1}(\mathbb{R})$, doing integration by parts,

 $\displaystyle\left\langle f,K_{x}\right\rangle_{H^{1}}$ $\displaystyle=\int_{-\infty}^{\infty}fK_{x}d\lambda+\int_{-\infty}^{x}f^{% \prime}(y)K_{x}(y)d\lambda(y)-\int_{x}^{\infty}f^{\prime}(y)K_{x}(y)d\lambda(y)$ $\displaystyle=\int_{-\infty}^{\infty}fK_{x}d\lambda+f(x)K_{x}(x)-\int_{-\infty% }^{x}f(y)K_{x}^{\prime}(y)d\lambda(y)$ $\displaystyle+f(x)K_{x}(x)+\int_{x}^{\infty}f(y)K_{x}^{\prime}(y)d\lambda(y)$ $\displaystyle=\int_{-\infty}^{\infty}fK_{x}d\lambda+\frac{1}{2}f(x)-\int_{-% \infty}^{x}f(y)K_{x}(y)d\lambda(y)$ $\displaystyle+\frac{1}{2}f(x)-\int_{x}^{\infty}f(y)K_{x}(y)d\lambda(y)$ $\displaystyle=f(x)$ $\displaystyle=T_{x}f.$

This shows that $H^{1}(\mathbb{R})$ is a reproducing kernel Hilbert space. We calculate, for $x,

 $\displaystyle\left\langle T_{x},T_{y}\right\rangle_{H^{1}}$ $\displaystyle=\int_{-\infty}^{x}K_{x}K_{y}d\lambda+\int_{x}^{y}K_{x}K_{y}d% \lambda+\int_{y}^{\infty}K_{x}K_{y}d\lambda$ $\displaystyle+\int_{-\infty}^{x}K_{x}K_{y}d\lambda-\int_{x}^{y}K_{x}K_{y}d% \lambda+\int_{y}^{\infty}K_{x}K_{y}d\lambda$ $\displaystyle=4\cdot\frac{1}{8}\exp(x-y)$ $\displaystyle=K(x,y).$

This shows that $K(x,y)=\frac{1}{2}\exp(-|x-y|)$ is the reproducing kernel of $H^{1}(\mathbb{R})$.77 7 cf. Alain Berlinet and Christine Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics, pp. 8–9, Example 5.