Real reproducing kernel Hilbert spaces

1 Reproducing kernels

We shall often speak about functions $F:X\times X\to ℝ$, where $X$ is a nonempty set. For $x\in X$, we define $F_x:X\to ℝ$ by $F_x(y)=F(x,y)$ and for $y\in X$ we define $F^y:X\to ℝ$ by $F^y(x)=F(x,y)$. $F$ is said to be symmetric if $F(x,y)=F(y,x)$ for all $x,y\in X$ and positive-definite if for any $x_1,\mathrm{\dots },x_n\in X$ and $c_1,\mathrm{\dots },c_n\in ℝ$ it holds that

$$\sum _{1\le i,j\le n}{c}_i{c}_{j}F(x_i,x_j)\ge 0.$$

Lemma 1.

If $F\mathrm{:}X\mathrm{\times }X\mathrm{\to }\mathrm{R}$ is symmetric and positive-definite then

$$F{(x,y)}^2\le F(x,x)F(y,y),x,y\in X.$$

Proof.

For $\alpha ,\beta \in ℝ$ define

$C(\alpha ,\beta )$ $={\alpha }^{2}F(x,x)+\alpha \beta F(x,y)+\beta \alpha F(y,x)+{\beta }^{2}F(y,y)$

$={\alpha }^{2}F(x,x)+2\alpha \beta F(x,y)+{\beta }^{2}F(y,y),$

which is $\ge 0$. Let

$P(\alpha )$ $=C(\alpha ,F(x,y))$

$={\alpha }^{2}F(x,x)+2\alpha F{(x,y)}^2+F{(x,y)}^{2}F(y,y),$

which is $\ge 0$. In the case $F(x,x)=0$, the fact that $P\ge 0$ implies that $F(x,y)=0$. In the case $F(x,y)\ne 0$, $P(\alpha )$ is a quadratic polynomial and because $P\ge 0$ it follows that the discriminant of $P$ is $\le 0$:

$$4F{(x,y)}^4-4\cdot F(x,x)\cdot F{(x,y)}^{2}F(y,y)\le 0.$$

That is, $F{(x,y)}^4\le F{(x,y)}^{2}F(x,x)F(y,y)$, and this implies that $F{(x,y)}^2\le F(x,x)F(y,y)$. ∎

A real reproducing kernel Hilbert space is a Hilbert space $H$ contained in ${ℝ}^X$, where $X$ is a nonempty set, such that for each $x\in X$ the map ${\mathrm{\Lambda }}_{x}f=f(x)$ is continuous $H\to ℝ$. In this note we speak always about real Hilbert spaces.

Let $H\subset {ℝ}^X$ be a reproducing kernel Hilbert space. Because $H$ is a Hilbert space, the Riesz representation theorem states that $\mathrm{\Phi }:H\to H^{*}$ defined by

$$(\mathrm{\Phi }g)(f)={⟨f,g⟩}_H,g,f\in H$$

is an isometric isomorphism. Because $H$ is a reproducing kernel Hilbert space, ${\mathrm{\Lambda }}_x\in H^{*}$ for each $x\in X$ and we define $T_x={\mathrm{\Phi }}^{-1}{\mathrm{\Lambda }}_x\in H$, which satisfies

$$f(x)={\mathrm{\Lambda }}_x(f)={⟨f,T_x⟩}_H,f\in H.$$

In particular, because $T_x\in H$, for $y\in X$ it holds that

$$T_x(y)={\mathrm{\Lambda }}_y(T_x)={⟨T_x,T_y⟩}_H.$$

Define $K:X\times X\to ℝ$ by

$$K(x,y)={⟨T_x,T_y⟩}_H,$$

called the reproducing kernel of $H$. For $x,y\in X$,

$$T_x(y)={⟨T_x,T_y⟩}_H=K(x,y)=K_x(y),$$

which means that $T_x=K_x$.

A reproducing kernel is symmetric and positive-definite:

$$K(x,y)={⟨T_x,T_y⟩}_H={⟨T_y,T_x⟩}_H=K(y,x)$$

and

${\displaystyle \sum _{1\le i,j\le n}}{c}_i{c}_{j}K(x_i,x_j)$ $={\displaystyle \sum _{1\le i,j\le n}}{⟨c_i{T}_{x_i},c_j{T}_{x_j}⟩}_H$

$={⟨{\displaystyle \sum _{1\le i\le n}}{c}_i{T}_{x_i},{\displaystyle \sum _{1\le j\le n}}{c}_j{T}_{x_j}⟩}_H$

$\ge 0.$

Lemma 2.

If $E$ is an orthonormal basis for a reproducing kernel Hilbert space $H\mathrm{\subset }{\mathrm{R}}^X$ with reproducing kernel $K\mathrm{:}X\mathrm{\times }X\mathrm{\to }\mathrm{R}$, then

$$K(x,y)=\sum _{e\in E}e(x)e(y),x,y\in X.$$

Proof.

Because $E$ is an orthonormal basis for $H$, Parseval’s identity tell us

$${⟨T_x,T_y⟩}_H=\sum _{e\in E}⟨T_x,e⟩⟨T_y,e⟩=\sum _{e\in E}⟨e,T_x⟩⟨e,T_y⟩=\sum _{e\in E}e(x)e(y).$$

∎

If $H\subset {ℝ}^X$ is a reproducing kernel Hilbert space with reproducing kernel $K:X\times X\to ℝ$ and $V$ is a closed linear subspace of $H$, then $V$ is itself a reproducing kernel Hilbert space, with some reproducing kernel $G:X\times X\to ℝ$. The following theorem expresses $G$ in terms of $K$.

Theorem 3.

Let $H\mathrm{\subset }{\mathrm{R}}^X$ be a reproducing kernel Hilbert space with reproducing kernel $K\mathrm{:}X\mathrm{\times }X\mathrm{\to }\mathrm{R}$, let $V$ be a closed linear subspace of $H$ with reproducing kernel $G\mathrm{:}X\mathrm{\times }X\mathrm{\to }\mathrm{R}$, and let $P_V\mathrm{:}H\mathrm{\to }V$ be the projection onto $V$. Then

$$G_x=P_V{K}_x,x\in X.$$

Proof.

$H=V\oplus V^{⟂}$, thus for $f\in H$ there are unique $g\in V,h\in V^{⟂}$ such that $f=g+h$, and $P_{V}f=g$. Then $f-P_{V}f\in V^{⟂}$. Therefore for $y\in X$, as $G_y\in V$ it holds that

$${⟨f,G_y⟩}_H={⟨f-P_{V}f+P_{V}f,G_y⟩}_H={⟨P_{V}f,G_y⟩}_H=(P_{V}f)(y).$$

In particular, for $x,y\in X$ and $f=K_x$,

$$(P_V{K}_x)(y)={⟨K_x,G_y⟩}_H={⟨G_y,T_x⟩}_H=G_y(x)=G(y,x)=G(x,y)=G_x(y),$$

which means that $P_V{K}_x=G_x$, proving the claim. ∎

The Moore-Aronszajn theorem states that if $X$ is a nonempty set and $K:X\times X\to ℝ$ is a symmetric and positive-definite function, then there is a unique reproducing kernel Hilbert space $H\subset {ℝ}^X$ for which $K$ is the reproducing kernel.

We now prove that given a symmetric positive-definite kernel there is a unique reproducing Hilbert space for which it is the reproducing kernel.

2 Sobolev spaces on $[0,T]$

Let $f\in {ℝ}^{[0,T]}$. The following are equivalent:

1. $f$ is absolutely continuous.

2. $f$ is differentiable at almost all $t\in [0,T]$, $f^{\prime }\in L^1$, and

   $$f(t)=f(0)+{\int }_0^t{f}^{\prime }(s)𝑑s,t\in [0,T].$$

3. There is some $g\in L^1$ such that

   $$f(t)=f(0)+{\int }_0^{t}g(s)𝑑s,t\in [0,T].$$

In particular, if $f$ is absolutely continuous and $f^{\prime }=0$ almost everywhere then ${\int }_0^t{f}^{\prime }(s)𝑑s=0$ and so $f(t)=f(0)$ for all $t\in [0,T]$. That is, if $f$ is absolutely continuous and $f^{\prime }=0$ almost everywhere then $f$ is constant.

Let $H$ be the set of those absolutely continuous functions $f\in {ℝ}^{[0,T]}$ such that $f(0)=0$ and $f^{\prime }\in L^2$. For $f,g\in H$ define

$${⟨f,g⟩}_H={\int }_0^T{f}^{\prime }(s)g^{\prime }(s)𝑑s.$$

If ${\parallel f\parallel }_H=0$ then ${\int }_0^T{f}^{\prime }{(s)}^2𝑑s=0$, which implies that $f^{\prime }=0$ almost everywhere and hence that $f$ is constant, and therefore $f=0$. Thus ${⟨\cdot ,\cdot ⟩}_H$ is indeed an inner product on $H$.

If $f_n$ is a Cauchy sequence in $H$ then $f_n^{\prime }$ is a Cauchy sequence in $L^2$ and hence converges to some $g\in L^2$. Then the function $f\in {ℝ}^{[0,T]}$ defined by

$$f(t)={\int }_0^{t}g(s)𝑑s,t\in [0,T],$$

is absolutely continuous, $f(0)=0$, and satisfies $f^{\prime }=g$ almost everywhere, which shows that $f\in H$. Then $f_n\to f$ in $H$, which proves that $H$ is a Hilbert space. For $t\in [0,T]$, by the Cauchy-Schwarz inequality,

$${|f(t)|}^2={\left|{\int }_0^t{f}^{\prime }(s)𝑑s\right|}^2\le {\left|{\int }_0^T{f}^{\prime }(s)𝑑s\right|}^2\le T{\int }_0^T{f}^{\prime }{(s)}^2𝑑s=T{\parallel f\parallel }_H^2,$$

i.e. $|L_{t}f|\le T^{1/2}{\parallel f\parallel }_H$, which shows that $L_t\in H^{*}$. Therefore $H$ is a reproducing kernel Hilbert space.

For $a\in [0,T]$ define $h_a:[0,T]\to ℝ$ by $h_a(s)=1_{[0,a]}(s)$, which belongs to $L^2$, and define $g_a:[0,T]\to ℝ$ by

$$g_a(t)={\int }_0^t{h}_a(s)𝑑s=\min (t,a),$$

which belongs to $H$. For $f\in H$,

$${⟨f,g_a⟩}_H={\int }_0^T{f}^{\prime }(s)g_a^{\prime }(s)𝑑s={\int }_0^T{f}^{\prime }(s)1_{[0,a]}(s)𝑑s={\int }_0^a{f}^{\prime }(s)𝑑s=f(a).$$

This means that $K_a=g_a$. For $a,b\in [0,T]$,

$${⟨K_a,K_b⟩}_H={\int }_0^T{g}_a^{\prime }(s)g_b^{\prime }(s)𝑑s={\int }_0^T{1}_{[0,a]}(s)1_{[0,b]}(s)𝑑s={\int }_0^T{1}_{[0,\min (a,b)]}(s)𝑑s.$$

That is, the reproducing kernel of $H$ is $K:[0,T]\times [0,T]\to ℝ$,

$$K(a,b)={⟨K_a,K_b⟩}_H=\min (a,b).$$

3 Sobolev spaces on $ℝ$

Let $\lambda$ be Lebesgue measure on $ℝ$. Let ${ℒ}^2(\lambda )$ be the collection of Borel measurable functions $f:ℝ\to ℝ$ such that ${|f|}^2$ is integrable, and let $L^2(\lambda )$ be the Hilbert space of equivalence classes of elements of ${ℒ}^2(\lambda )$ where $f\sim g$ when $f=g$ almost everywhere, with

$${⟨f,g⟩}_{L^2}={\int }_{ℝ}fg𝑑\lambda .$$

Let $H^1(ℝ)$ be the set of locally absolutely continuous functions $f:ℝ\to ℝ$ such that $f,f^{\prime }\in L^2(\lambda )$. This is a Hilbert space with the inner product

$${⟨f,g⟩}_{H^1}={⟨f,g⟩}_{L^2}+{⟨f^{\prime },g^{\prime }⟩}_{L^2}.$$

Define $K:ℝ\times ℝ\to ℝ$ by

$$K(x,y)=\frac{1}{2}\exp (-|x-y|),x,y\in ℝ.$$

Let $x\in ℝ$. For , $K_x^{\prime }(y)=K_x(y)$ and for $y>x$, $K_x^{\prime }(y)=-K_x(y)$, which shows that $K_x\in H^1(ℝ)$. For $f\in H^1(ℝ)$, doing integration by parts,

${⟨f,K_x⟩}_{H^1}$ $={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}f{K}_x𝑑\lambda +{\displaystyle {\int }_{-\mathrm{\infty }}^x}{f}^{\prime }(y)K_x(y)𝑑\lambda (y)-{\displaystyle {\int }_x^{\mathrm{\infty }}}{f}^{\prime }(y)K_x(y)𝑑\lambda (y)$

$={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}f{K}_x𝑑\lambda +f(x)K_x(x)-{\displaystyle {\int }_{-\mathrm{\infty }}^x}f(y)K_x^{\prime }(y)𝑑\lambda (y)$

$+f(x)K_x(x)+{\displaystyle {\int }_x^{\mathrm{\infty }}}f(y)K_x^{\prime }(y)𝑑\lambda (y)$

$={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}f{K}_x𝑑\lambda +{\displaystyle \frac{1}{2}}f(x)-{\displaystyle {\int }_{-\mathrm{\infty }}^x}f(y)K_x(y)𝑑\lambda (y)$

$+{\displaystyle \frac{1}{2}}f(x)-{\displaystyle {\int }_x^{\mathrm{\infty }}}f(y)K_x(y)𝑑\lambda (y)$

$=f(x)$

$=T_{x}f.$

This shows that $H^1(ℝ)$ is a reproducing kernel Hilbert space. We calculate, for ,

${⟨T_x,T_y⟩}_{H^1}$ $={\displaystyle {\int }_{-\mathrm{\infty }}^x}{K}_x{K}_y𝑑\lambda +{\displaystyle {\int }_x^y}{K}_x{K}_y𝑑\lambda +{\displaystyle {\int }_y^{\mathrm{\infty }}}{K}_x{K}_y𝑑\lambda$

$+{\displaystyle {\int }_{-\mathrm{\infty }}^x}{K}_x{K}_y𝑑\lambda -{\displaystyle {\int }_x^y}{K}_x{K}_y𝑑\lambda +{\displaystyle {\int }_y^{\mathrm{\infty }}}{K}_x{K}_y𝑑\lambda$

$=4\cdot {\displaystyle \frac{1}{8}}\exp (x-y)$

$=K(x,y).$

This shows that $K(x,y)=\frac{1}{2}\exp (-|x-y|)$ is the reproducing kernel of $H^1(ℝ)$.