Fréchet derivatives and Gâteaux derivatives

1 Introduction

In this note all vector spaces are real. If $X$ and $Y$ are normed spaces, we denote by $ℬ(X,Y)$ the set of bounded linear maps $X\to Y$, and write $ℬ(X)=ℬ(X,X)$. $ℬ(X,Y)$ is a normed space with the operator norm.

2 Remainders

If $X$ and $Y$ are normed spaces, let $o(X,Y)$ be the set of all maps $r:X\to Y$ for which there is some map $\alpha :X\to Y$ satisfying:

• $r(x)=\parallel x\parallel \alpha (x)$ for all $x\in X$,

• $\alpha (0)=0$,

• $\alpha$ is continuous at $0$.

Following Penot, we call elements of $o(X,Y)$ remainders. It is immediate that $o(X,Y)$ is a vector space.

If $X$ and $Y$ are normed spaces, if $f:X\to Y$ is a function, and if $x_0\in X$, we say that $f$ is stable at $x_{\mathrm{0}}$ if there is some $ϵ>0$ and some $c>0$ such that $\parallel x-x_0\parallel \le ϵ$ implies that $\parallel f(x-x_0)\parallel \le c\parallel x-x_0\parallel$. If $T:X\to Y$ is a bounded linear map, then $\parallel Tx\parallel \le \parallel T\parallel \parallel x\parallel$ for all $x\in X$, and thus a bounded linear map is stable at $0$. The following lemma shows that the composition of a remainder with a function that is stable at $0$ is a remainder.

Lemma 1.

Let $X,Y$ be normed spaces and let $r\in o(X,Y)$. If $W$ is a normed space and $f:W\to X$ is stable at 0, then $r\circ f\in o(W,Y)$. If $Z$ is a normed space and $g:Y\to Z$ is stable at 0, then $g\circ r\in o(X,Z)$.

Proof.

$r\in o(X,Y)$ means that there is some $\alpha :X\to Y$ satisfying $r(x)=\parallel x\parallel \alpha (x)$ for all $x\in X$, that takes the value $0$ at $0$, and that is continuous at $0$. As $f$ is stable at $0$, there is some $ϵ>0$ and some $c>0$ for which $\parallel w\parallel \le ϵ$ implies that $\parallel f(w)\parallel \le c\parallel w\parallel$. Define $\beta :W\to Y$ by

for which we have

$$(r\circ f)(w)=\parallel w\parallel \beta (w),w\in W.$$

If $\parallel w\parallel \le ϵ$, then $\parallel \beta (w)\parallel \le c\parallel \alpha (f(w))\parallel$. But $f(w)\to 0$ as $w\to 0$, and because $\alpha$ is continuous at $0$ we get that $\alpha (f(w))\to \alpha (0)=0$ as $w\to 0$. So the above inequality gives us $\beta (w)\to 0$ as $w\to 0$. As $\beta (0)=0$, the function $\beta :W\to Y$ is continuous at $0$, and therefore $r\circ f$ is remainder.

As $g$ is stable at $0$, there is some $ϵ>0$ and some $c>0$ for which $\parallel y\parallel \le ϵ$ implies that $\parallel g(y)\parallel \le c\parallel y\parallel$. Define $\gamma :X\to Z$ by

For all $x\in X$,

$$(g\circ r)(x)=g(\parallel x\parallel \alpha (x))=\parallel x\parallel \gamma (x).$$

Since $\alpha (0)=0$ and $\alpha$ is continuous at $0$, there is some $\delta >0$ such that $\parallel x\parallel \le \delta$ implies that $\parallel \alpha (x)\parallel \le ϵ$. Therefore, if $\parallel x\parallel \le \delta \wedge 1$ then

$$\parallel g(\parallel x\parallel \alpha (x))\parallel \le c\parallel x\parallel \parallel \alpha (x)\parallel \le c\parallel x\parallel ϵ,$$

and hence if $\parallel x\parallel \le \delta \wedge 1$ then $\parallel \gamma (x)\parallel \le cϵ$. This shows that $\gamma (x)\to 0$ as $x\to 0$, and since $\gamma (0)=0$ the function $\gamma :X\to Z$ is continuous at $0$, showing that $g\circ r$ is a remainder. ∎

If $Y_1,\mathrm{\dots },Y_n$ are normed spaces where $Y_k$ has norm $\parallel \cdot {\parallel }_k$, then $\parallel (y_1,\mathrm{\dots },y_n)\parallel ={\max }_{1\le k\le n}{\parallel y_k\parallel }_k$ is a norm on ${\prod }_{k=1}^n{Y}_k$, and one can prove that the topology induced by this norm is the product topology.

Lemma 2.

If $X$ and $Y_1,\mathrm{\dots },Y_n$ are normed spaces, then a function $r:X\to {\prod }_{k=1}^n{Y}_k$ is a remainder if and only if each of $r_k:X\to Y_k$ are remainders, $1\le k\le n$, where $r(x)=(r_1(x),\mathrm{\dots },r_n(x))$ for all $x\in X$.

Proof.

Suppose that there is some function $\alpha :X\to {\prod }_{k=1}^n{Y}_k$ such that $r(x)=\parallel x\parallel \alpha (x)$ for all $x\in X$. With $\alpha (x)=({\alpha }_1(x),\mathrm{\dots },{\alpha }_n(x))$, we have

$$r_k(x)=\parallel x\parallel {\alpha }_k(x),x\in X.$$

Because $\alpha (x)\to 0$ as $x\to 0$, for each $k$ we have ${\alpha }_k(x)\to 0$ as $x\to 0$, which shows that $r_k$ is a remainder.

Suppose that each $r_k$ is a remainder. Thus, for each $k$ there is a function ${\alpha }_k:X\to Y_k$ satisfying $r_k(x)=\parallel x\parallel {\alpha }_k(x)$ for all $x\in X$ and ${\alpha }_k(x)\to 0$ as $x\to 0$. Then the function $\alpha :X\to {\prod }_{k=1}^n{Y}_k$ defined by $\alpha (x)=({\alpha }_1(x),\mathrm{\dots },{\alpha }_n(x))$ satisfies $r(x)=\parallel x\parallel \alpha (x)$. Because ${\alpha }_k(x)\to 0$ as $x\to 0$ for each of the finitely many $k$, $1\le k\le n$, we have $\alpha (x)\to 0$ as $x\to 0$. ∎

3 Definition and uniqueness of Fréchet derivative

Suppose that $X$ and $Y$ are normed spaces, that $U$ is an open subset of $X$, and that $x_0\in U$. A function $f:U\to Y$ is said to be Fréchet differentiable at $x_{\mathrm{0}}$ if there is some $L\in ℬ(X,Y)$ and some $r\in o(X,Y)$ such that

$$f(x)=f(x_0)+L(x-x_0)+r(x-x_0),x\in U.$$ (1)

Suppose there are bounded linear maps $L_1,L_2$ and remainders $r_1,r_2$ that satisfy the above. Writing $r_1(x)=\parallel x\parallel {\alpha }_1(x)$ and $r_2(x)=\parallel x\parallel {\alpha }_2(x)$ for all $x\in X$, we have

$$L_1(x-x_0)+\parallel x-x_0\parallel {\alpha }_1(x-x_0)=L_2(x-x_0)+\parallel x-x_0\parallel {\alpha }_2(x-x_0),x\in U,$$

i.e.,

$$L_1(x-x_0)-L_2(x-x_0)=\parallel x-x_0\parallel ({\alpha }_2(x-x_0)-{\alpha }_1(x-x_0)),x\in U.$$

For $x\in X$, there is some $h>0$ such that for all $|t|\le h$ we have $x_0+tx\in U$, and then

$$L_1(tx)-L_2(tx)=\parallel tx\parallel ({\alpha }_2(tx)-{\alpha }_1(tx)),$$

hence, for ,

$$L_1(x)-L_2(x)=\parallel x\parallel ({\alpha }_2(tx)-{\alpha }_1(tx)).$$

But ${\alpha }_2(tx)-{\alpha }_1(tx)\to 0$ as $t\to 0$, which implies that $L_1(x)-L_2(x)=0$. As this is true for all $x\in X$, we have $L_1=L_2$ and then $r_1=r_2$. If $f$ is Fréchet differentiable at $x_0$, the bounded linear map $L$ in (1) is called the Fréchet derivative of $f$ at $x_{\mathrm{0}}$, and we define $Df(x_0)=L$. Thus,

$$f(x)=f(x_0)+Df(x_0)(x-x_0)+r(x-x_0),x\in U.$$

If $U_0$ is the set of those points in $U$ at which $f$ is Fréchet differentiable, then $Df:U_0\to ℬ(X,Y)$.

Suppose that $X$ and $Y$ are normed spaces and that $U$ is an open subset of $X$. We denote by $C^1(U,Y)$ the set of functions $f:U\to Y$ that are Fréchet differentiable at each point in $U$ and for which the function $Df:U\to ℬ(X,Y)$ is continuous. We say that an element of $C^1(U,Y)$ is continuously differentiable. We denote by $C^2(U,Y)$ those elements $f$ of $C^1(U,Y)$ such that

$$Df\in C^1(U,ℬ(X,Y));$$

that is, $C^2(U,Y)$ are those $f\in C^1(U,Y)$ such that the function $Df:U\to ℬ(X,Y)$ is Fréchet differentiable at each point in $U$ and such that the function

$$D(Df):U\to ℬ(X,ℬ(X,Y))$$

is continuous.

The following theorem characterizes continuously differentiable functions ${ℝ}^n\to {ℝ}^m$.

Theorem 3.

Suppose that $f:{ℝ}^n\to {ℝ}^m$ is Fréchet differentiable at each point in ${ℝ}^n$, and write

$$f=(f_1,\mathrm{\dots },f_m).$$

$f\in C^1({ℝ}^n,{ℝ}^m)$ if and only if for each $1\le i\le m$ and $1\le j\le n$ the function

$$\frac{\partial f_i}{\partial x_j}:{ℝ}^n\to ℝ$$

is continuous.

4 Properties of the Fréchet derivative

If $f:X\to Y$ is Fréchet differentiable at $x_0$, then because a bounded linear map is continuous and in particular continuous at $0$, and because a remainder is continuous at $0$, we get that $f$ is continuous at $x_0$.

We now prove that Fréchet differentiation at a point is linear.

Lemma 4 (Linearity).

Let $X$ and $Y$ be normed spaces, let $U$ be an open subset of $X$ and let $x_0\in U$. If $f_1,f_2:U\to Y$ are both Fréchet differentiable at $x_0$ and if $\alpha \in ℝ$, then $\alpha f_1+f_2$ is Fréchet differentiable at $x_0$ and

$$D(\alpha f_1+f_2)(x_0)=\alpha D{f}_1(x_0)+D{f}_2(x_0).$$

Proof.

There are remainders $r_1,r_2\in o(X,Y)$ such that

$$f_1(x)=f_1(x_0)+D{f}_1(x_0)(x-x_0)+r_1(x-x_0),x\in U,$$

and

$$f_2(x)=f_2(x_0)+D{f}_2(x_0)(x-x_0)+r_2(x-x_0),x\in U.$$

Then for all $x\in U$,

$(\alpha f_1+f_2)(x)-(\alpha f_1+f_2)(x_0)$ $=$ $\alpha f_1(x)-\alpha f_1(x_0)+f_2(x)-f_2(x_0)$

$=$ $\alpha D{f}_1(x_0)(x-x_0)+\alpha r_1(x-x_0)$

$+D{f}_2(x_0)(x-x_0)+r_2(x-x_0)$

$=$ $(\alpha D{f}_1(x_0)+D{f}_2(x_0))(x-x_0)$

$+(\alpha r_1+r_2)(x-x_0),$

and $\alpha r_1+r_2\in o(X,Y)$. ∎

The following lemma gives an alternate characterization of a function being Fréchet differentiable at a point.

Lemma 5.

Suppose that $X$ and $Y$ are normed space, that $U$ is an open subset of $X$, and that $x_0\in U$. A function $f:U\to Y$ is Fréchet differentiable at $x_0$ if and only if there is some function $F:U\to ℬ(X,Y)$ that is continuous at $x_0$ and for which

$$f(x)-f(x_0)=F(x)(x-x_0),x\in U.$$

Proof.

Suppose that there is a function $F:U\to ℬ(X,Y)$ that is continuous at $x_0$ and that satisfies $f(x)-f(x_0)=F(x)(x-x_0)$ for all $x\in U$. Then, for $x\in U$,

$f(x)-f(x_0)$ $=$ $F(x)(x-x_0)-F(x_0)(x-x_0)+F(x_0)(x-x_0)$

$=$ $F(x_0)(x-x_0)+r(x-x_0),$

where $r:X\to Y$ is defined by

We further define

with which $r(x)=\parallel x\parallel \alpha (x)$ for all $x\in X$. To prove that $r$ is a remainder it suffices to prove that $\alpha (x)\to 0$ as $x\to 0$. Let $ϵ>0$. That $F:U\to ℬ(X,Y)$ is continuous at $x_0$ tells us that there is some $\delta >0$ for which implies that and hence

Therefore, if then , which establishes that $r$ is a remainder and therefore that $f$ is Fréchet differentiable at $x_0$, with Fréchet derivative $Df(x_0)=F(x_0)$.

Suppose that $f$ is Fréchet differentiable at $x_0$: there is some $r\in o(X,Y)$ such that

$$f(x)=f(x_0)+Df(x_0)(x-x_0)+r(x-x_0),x\in U,$$

where $Df(x_0)\in ℬ(X,Y)$. As $r$ is a remainder, there is some $\alpha :X\to Y$ satisfying $r(x)=\parallel x\parallel \alpha (x)$ for all $x\in X$, and such that $\alpha (0)=0$ and $\alpha (x)\to 0$ as $x\to 0$. For each $x\in X$, by the Hahn-Banach extension theorem there is some ${\lambda }_x\in X^{*}$ such that ${\lambda }_{x}x=\parallel x\parallel$ and $|{\lambda }_{x}v|\le \parallel v\parallel$ for all $v\in X$. Thus,

$$r(x)=({\lambda }_{x}x)\alpha (x),x\in X.$$

Define $F:U\to ℬ(X,Y)$ by

$$F(x)=Df(x_0)+({\lambda }_{x-x_0})\alpha (x-x_0),$$

i.e. for $x\in U$ and $v\in X$,

$$F(x)(v)=Df(x_0)(v)+({\lambda }_{x-x_0}v)\alpha (x-x_0)\in Y.$$

Then for $x\in U$,

$$r(x-x_0)=({\lambda }_{x-x_0}(x-x_0))\alpha (x-x_0)=F(x)(x-x_0)-Df(x_0)(x-x_0),$$

and hence

$$f(x)=f(x_0)+F(x)(x-x_0),x\in U.$$

To complete the proof it suffices to prove that $F$ is continuous at $x_0$. But both ${\lambda }_0=0$ and $\alpha (0)=0$ so $F(x_0)=Df(x_0)$, and for $x\in U$ and $v\in X$,

$\parallel (F(x)-F(x_0))(v)\parallel$ $=$ $\parallel ({\lambda }_{x-x_0}v)\alpha (x-x_0)\parallel$

$=$ $|{\lambda }_{x-x_0}v|\parallel \alpha (x-x_0)\parallel$

$\le$ $\parallel v\parallel \parallel \alpha (x-x_0)\parallel ,$

so $\parallel F(x)-F(x_0)\parallel \le \parallel \alpha (x-x_0)\parallel$. From this and the fact that $\alpha (0)=0$ and $\alpha (x)\to 0$ as $x\to 0$ we get that $F$ is continuous at $x_0$, completing the proof. ∎

We now prove the chain rule for Fréchet derivatives.

Theorem 6 (Chain rule).

Suppose that $X,Y,Z$ are normed spaces and that $U$ and $V$ are open subsets of $X$ and $Y$ respectively. If $f:U\to Y$ satisfies $f(U)\subseteq V$ and is Fréchet differentiable at $x_0$ and if $g:V\to Z$ is Fréchet differentiable at $f(x_0)$, then $g\circ f:U\to Z$ is Fréchet differentiable at $x_0$, and its Fréchet derivative at $x_0$ is

$$D(g\circ f)(x_0)=Dg(f(x_0))\circ Df(x_0).$$

Proof.

Write $y_0=f(x_0)$, $L_1=Df(x_0)$, and $L_2=Dg(y_0)$. Because $f$ is Fréchet differentiable at $x_0$, there is some $r_1\in o(X,Y)$ such that

$$f(x)=f(x_0)+L_1(x-x_0)+r_1(x-x_0),x\in U,$$

and because $g$ is Fréchet differentiable at $y_0$ there is some $r_2\in o(Y,Z)$ such that

$$g(y)=g(y_0)+L_2(y-y_0)+r_2(y-y_0),y\in V.$$

For all $x\in U$ we have $f(x)\in V$, and using the above formulas,

$g(f(x))$ $=g(y_0)+L_2(f(x)-y_0)+r_2(f(x)-y_0)$

$=g(y_0)+L_2\left(L_1(x-x_0)+r_1(x-x_0)\right)+r_2\left(L_1(x-x_0)+r_1(x-x_0)\right)$

$=g(y_0)+L_2(L_1(x-x_0))+L_2(r_1(x-x_0))+r_2\left(L_1(x-x_0)+r_1(x-x_0)\right).$

Define $r_3:X\to Z$ by $r_3(x)=r_2(L_{1}x+r_1(x))$, and fix any $c>\parallel L_1\parallel$. Writing $r_1(x)=\parallel x\parallel {\alpha }_1(x)$, the fact that $\alpha (0)=0$ and that $\alpha$ is continuous at $0$ gives us that there is some $\delta >0$ such that if then , and hence if then $\parallel r_1(x)\parallel \le (c-\parallel L_1\parallel )\parallel x\parallel$. Then, implies that

$$\parallel L_{1}x+r_1(x)\parallel \le \parallel L_{1}x\parallel +\parallel r_1(x)\parallel \le \parallel L_1\parallel \parallel x\parallel +(c-\parallel L_1\parallel )\parallel x\parallel =c\parallel x\parallel .$$

This shows that $x\mapsto L_{1}x+r_1(x)$ is stable at $0$ and so by Lemma 1 that $r_3\in o(X,Z)$. Then, $r:X\to Z$ defined by $r=L_1\circ r_1+r_3$ is a sum of two remainders and so is itself a remainder, and we have

$$g\circ f(x)=g\circ f(x_0)+L_2\circ L_1(x-x_0)+r(x-x_0),x\in U.$$

But $L_1\in ℬ(X,Y)$ and $L_2\in ℬ(Y,Z)$, so $L_2\circ L_1\in ℬ(X,Z)$. This shows that $g\circ f$ is Fréchet differentiable at $x_0$ and that its Fréchet derivative at $x_0$ is

$$L_2\circ L_1=Dg(y_0)\circ Df(x_0)=Dg(f(x_0))\circ Df(x_0).$$

∎

The following is the product rule for Fréchet derivatives. By $f_1\cdot f_2$ we mean the function $x\mapsto f_1(x)f_2(x)$.

Theorem 7 (Product rule).

Suppose that $X$ is a normed space, that $U$ is an open subset of $X$, that $f_1,f_2:U\to ℝ$ are functions, and that $x_0\in U$. If $f_1$ and $f_2$ are both Fréchet differentiable at $x_0$, then $f_1\cdot f_2$ is Fréchet differentiable at $x_0$, and its Fréchet derivative at $x_0$ is

$$D(f_1\cdot f_2)(x_0)=f_2(x_0)D{f}_1(x_0)+f_1(x_0)D{f}_2(x_0).$$

Proof.

There are $r_1,r_2\in o(X,ℝ)$ with which

$$f_1(x)=f_1(x_0)+D{f}_1(x_0)(x-x_0)+r_1(x-x_0),x\in U$$

and

$$f_2(x)=f_2(x_0)+D{f}_2(x_0)(x-x_0)+r_2(x-x_0),x\in U.$$

Multiplying the above two formulas,

$f_1(x)f_2(x)$ $=$ $f_1(x_0)f_2(x_0)+f_2(x_0)D{f}_1(x_0)(x-x_0)+f_1(x_0)D{f}_2(x_0)(x-x_0)$

$+D{f}_1(x_0)(x-x_0)D{f}_2(x_0)(x-x_0)+r_1(x-x_0)r_2(x-x_0)$

$+f_1(x_0)r_2(x-x_0)+r_2(x-x_0)D{f}_1(x_0)(x-x_0)$

$+f_2(x_0)r_1(x-x_0)+r_1(x-x_0)D{f}_2(x_0)(x-x_0).$

Define $r:X\to ℝ$ by

$r(x)$ $=$ $D{f}_1(x_0)xD{f}_2(x_0)x+r_1(x)r_2(x)+f_1(x_0)r_2(x)+r_2(x)D{f}_1(x_0)x$

$+f_2(x_0)r_1(x)+r_1(x)D{f}_2(x_0)x,$

for which we have, for $x\in U$,

$$f_1(x)f_2(x)=f_1(x_0)f_2(x_0)+f_2(x_0)D{f}_1(x_0)(x-x_0)+f_1(x_0)D{f}_2(x_0)(x-x_0)+r(x-x_0).$$

Therefore, to prove the claim it suffices to prove that $r\in o(X,ℝ)$. Define $\alpha :X\to ℝ$ by $\alpha (0)=0$ and $\alpha (x)=\frac{D{f}_1(x_0)xD{f}_2(x_0)x}{\parallel x\parallel }$ for $x\ne 0$. For $x\ne 0$,

$|\alpha (x)|$ $=$ ${\displaystyle \frac{|D{f}_1(x_0)x||D{f}_2(x_0)x|}{\parallel x\parallel }}$

$\le$ ${\displaystyle \frac{\parallel D{f}_1(x_0)\parallel \parallel x\parallel \parallel D{f}_2(x_0)\parallel \parallel x\parallel }{\parallel x\parallel }}$

$=$ $\parallel D{f}_1(x_0)\parallel \parallel D{f}_2(x_0)\parallel \parallel x\parallel .$

Thus $\alpha (x)\to 0$ as $x\to 0$, showing that the first term in the expression for $r$ belongs to $o(X,ℝ)$. Likewise, each of the other five terms in the expression for $r$ belongs to $o(X,ℝ)$, and hence $r\in o(X,ℝ)$, completing the proof. ∎

5 Dual spaces

If $X$ is a normed space, we denote by $X^{*}$ the set of bounded linear maps $X\to ℝ$, i.e. $X^{*}=ℬ(X,ℝ)$. $X^{*}$ is itself a normed space with the operator norm. If $X$ is a normed space, the dual pairing $⟨\cdot ,\cdot ⟩:X\times X^{*}\to ℝ$ is

$$⟨x,\psi ⟩=\psi (x),x\in X,\psi \in X^{*}.$$

If $U$ is an open subset of $X$ and if a function $f:U\to ℝ$ is Fréchet differentiable at $x_0\in U$, then $Df(x_0)$ is a bounded linear map $X\to ℝ$, and so belongs to $X^{*}$. If $U_0$ are those points in $U$ at which $f:U\to ℝ$ is Fréchet differentiable, then

$$Df:U_0\to X^{*}.$$

In the case that $X$ is a Hilbert space with inner product $⟨\cdot ,\cdot ⟩$, the Riesz representation theorem shows that $R:X\to X^{*}$ defined by $R(x)(y)=⟨y,x⟩$ is an isometric isomorphism. If $f:U\to ℝ$ is Fréchet differentiable at $x_0\in U$, then we define

$$\nabla f(x_0)=R^{-1}(Df(x_0)),$$

and call $\nabla f(x_0)\in X$ the gradient of $f$ at $x_{\mathrm{0}}$. With $U_0$ denoting the set of those points in $U$ at which $f$ is Fréchet differentiable,

$$\nabla f:U_0\to X.$$

(To define the gradient we merely used that $R$ is a bijection, but to prove properties of the gradient one uses that $R$ is an isometric isomorphism.)

Example. Let $X$ be a Hilbert space, $A\in ℬ(X)$, $v\in X$, and define

$$f(x)=⟨Ax,x⟩-⟨x,v⟩,x\in X.$$

For all $x_0,x\in X$ we have, because the inner product of a real Hilbert space is symmetric,

$f(x)-f(x_0)$ $=$ $⟨Ax,x⟩-⟨x,v⟩-⟨A{x}_0,x_0⟩+⟨x_0,v⟩$

$=$ $⟨Ax,x⟩-⟨A{x}_0,x⟩+⟨A{x}_0,x⟩-⟨A{x}_0,x_0⟩-⟨x-x_0,v⟩$

$=$ $⟨A(x-x_0),x⟩+⟨A{x}_0,x-x_0⟩-⟨x-x_0,v⟩$

$=$ $⟨x-x_0,A^{*}x⟩+⟨x-x_0,A{x}_0⟩-⟨x-x_0,v⟩$

$=$ $⟨x-x_0,A^{*}x+A{x}_0-v⟩$

$=$ $⟨x-x_0,A^{*}x-A^{*}{x}_0+A^{*}{x}_0+A{x}_0-v⟩$

$=$ $⟨x-x_0,(A^{*}+A)x_0-v⟩+⟨x-x_0,A^{*}(x-x_0)⟩.$

With $Df(x_0)(x-x_0)=⟨x-x_0,(A^{*}+A)x_0-v⟩$, or $Df(x_0)(x)=⟨x,(A^{*}+A)x_0-v⟩$, we have that $f$ is Fréchet differentiable at each $x_0\in X$. Furthermore, its gradient at $x_0$ is

$$\nabla f(x_0)=(A^{*}+A)x_0-v.$$

For each $x_0\in X$, the function $f:X\to ℝ$ is Fréchet differentiable at $x_0$, and thus

$$Df:X\to X^{*},$$

and we can ask at what points $Df$ has a Fréchet derivative. For $x_0,x,y\in X$,

$(Df(x)-Df(x_0))(y)=$ $⟨y,(A^{*}+A)x-v⟩-⟨y,(A^{*}+A)x_0-v⟩$

$=$ $⟨y,(A^{*}+A)(x-x_0)⟩.$

For $D(Df)(x_0)(x-x_0)(y)=⟨y,(A^{*}+A)(x-x_0)⟩$, in other words with

$$D^{2}f(x_0)(x)(y)=D(Df)(x_0)(x)(y)=⟨y,(A^{*}+A)x⟩,$$

we have that $Df$ is Fréchet differentiable at each $x_0\in X$. Thus

$$D^{2}f:X\to ℬ(X,X^{*}).$$

Because $D^{2}f(x_0)$ does not depend on $x_0$, it is Fréchet differentiable at each point in $X$, with $D^{3}f(x_0)=0$ for all $x_0\in X$. Here $D^{3}f:X\to ℬ(X,ℬ(X,X^{*}))$.

6 Gâteaux derivatives

Let $X$ and $Y$ be normed spaces, let $U$ be an open subset of $X$, let $f:U\to Y$ be a function, and let $x_0\in U$. If there is some $T\in ℬ(X,Y)$ such that for all $v\in X$ we have

$$\underset{t\to 0}{lim}\frac{f(x_0+tv)-f(x_0)}{t}=Tv,$$ (2)

then we say that $f$ is Gâteaux differentiable at $x_{\mathrm{0}}$ and call $T$ the Gâteaux derivative of $f$ at $x_{\mathrm{0}}$. It is apparent that there is at most one $T\in ℬ(X,Y)$ that satisfies (2) for all $v\in X$. We write $f^{\prime }(x_0)=T$. Thus, $f^{\prime }$ is a map from the set of points in $U$ at which $f$ is Gâteaux differentiable to $ℬ(X,Y)$. If $V\subseteq U$ and $f$ is Gâteaux differentiable at each element of $V$, we say that $f$ is Gâteaux differentiable on $V$.

Example. Define $f:{ℝ}^2\to ℝ$ by $f(x_1,x_2)=\frac{x_1^4{x}_2}{x_1^6+x_2^3}$ for $(x_1,x_2)\ne (0,0)$ and $f(0,0)=0$. For $v=(v_1,v_2)\in {ℝ}^2$ and $t\ne 0$,

Hence, for any $v\in {ℝ}^2$, we have $\frac{f(0+tv)-f(0)}{t}\to 0$ as $t\to 0$. Therefore, $f$ is Gâteaux differentiable at $(0,0)$ and $f^{\prime }(0,0)v=0\in ℝ$ for all $v\in {ℝ}^2$, i.e. $f^{\prime }(0,0)=0$. However, for $(x_1,x_2)\ne (0,0)$,

$$f(x_1,x_1^2)=\frac{x_1^6}{x_1^6+x_1^6}=\frac{1}{2},$$

from which it follows that $f$ is not continuous at $(0,0)$. We stated in §4 that if a function is Fréchet differentiable at a point then it is continuous at that point, and so $f$ is not Fréchet differentiable at $(0,0)$. Thus, a function that is Gâteaux differentiable at a point need not be Fréchet differentiable at that point.

We prove that being Fréchet differentiable at a point implies being Gâteaux differentiable at the point, and that in this case the Gâteaux derivative is equal to the Fréchet derivative.

Theorem 8.

Suppose that $X$ and $Y$ are normed spaces, that $U$ is an open subset of $X$, that $f\in Y^U$, and that $x_0\in U$. If $f$ is Fréchet differentiable at $x_0$, then $f$ is Gâteaux differentiable at $x_0$ and $f^{\prime }(x_0)=Df(x_0)$.

Proof.

Because $f$ is Fréchet differentiable at $x_0$, there is some $r\in o(X,Y)$ for which

$$f(x)=f(x_0)+Df(x_0)(x-x_0)+r(x-x_0),x\in U.$$

For $v\in X$ and nonzero $t$ small enough that $x_0+tv\in U$,

$$\frac{f(x_0+tv)-f(x_0)}{t}=\frac{Df(x_0)(x_0+tv-x_0)+r(x_0+tv-x_0)}{t}=\frac{tDf(x_0)v+r(tv)}{t}.$$

Writing $r(x)=\parallel x\parallel \alpha (x)$,

$$\frac{f(x_0+tv)-f(x_0)}{t}=\frac{tDf(x_0)+\parallel tv\parallel \alpha (tv)}{t}=Df(x_0)v+\parallel v\parallel \alpha (tv).$$

Hence,

$$\underset{t\to 0}{lim}\frac{f(x_0+tv)-f(x_0)}{t}=Df(x_0)v.$$

This holds for all $v\in X$, and as $Df(x_0)\in ℬ(X,Y)$ we get that $f$ is Gâteaux differentiable at $x_0$ and that $f^{\prime }(x_0)=Df(x_0)$. ∎

If $X$ is a vector space and $u,v\in X$, let

$$[u,v]=\{(1-t)u+tv:0\le t\le 1\},$$

namely, the line segment joining $u$ and $v$. The following is a mean value theorem for Gâteaux derivatives.

Theorem 9 (Mean value theorem).

Let $X$ and $Y$ be normed spaces, let $U$ be an open subset of $X$, and let $f:U\to Y$ be Gâteaux differentiable on $U$. If $u,v\in U$ and $[u,v]\subset U$, then

$$\parallel f(u)-f(v)\parallel \le \underset{w\in [u,v]}{sup}\parallel f^{\prime }(w)\parallel \cdot \parallel u-v\parallel .$$

Proof.

If $f(u)=f(v)$ then immediately the claim is true. Otherwise, $f(v)-f(u)\ne 0$, and so by the Hahn-Banach extension theorem there is some $\psi \in Y^{*}$ satisfying $\psi (f(v)-f(u))=\parallel f(v)-f(u)\parallel$ and $\parallel \psi \parallel =1$. Define $h:[0,1]\to ℝ$ by

$$h(t)=⟨f((1-t)u+tv),\psi ⟩.$$

For and $\tau \ne 0$ satisfying $t+\tau \in [0,1]$, we have

${\displaystyle \frac{h(t+\tau )-h(t)}{\tau }}$ $=$ ${\displaystyle \frac{1}{\tau }}⟨f((1-t-\tau )u+(t+\tau )v),\psi ⟩-{\displaystyle \frac{1}{\tau }}⟨f((1-t)u+tv),\psi ⟩$

$=$ $⟨{\displaystyle \frac{f((1-t)u+tv+(v-u)\tau )-f((1-t)u+tv)}{\tau }},\psi ⟩.$

Because $f$ is Gâteaux differentiable at $(1-t)u+tv$,

$$\underset{\tau \to 0}{lim}\frac{f((1-t)u+tv+(v-u)\tau )-f((1-t)u+tv)}{\tau }=f^{\prime }((1-t)u+tv)(v-u),$$

so because $\psi$ is continuous,

$$\underset{\tau \to 0}{lim}\frac{h(t+\tau )-h(t)}{\tau }=⟨f^{\prime }((1-t)u+tv)(v-u),\psi ⟩,$$

which shows that $h$ is differentiable at $t$ and that

$$h^{\prime }(t)=⟨f^{\prime }((1-t)u+tv)(v-u),\psi ⟩.$$

$h:[0,1]\to ℝ$ is a composition of continuous functions so it is continuous. Applying the mean value theorem, there is some $\theta$, , for which

$$h^{\prime }(\theta )=h(1)-h(0).$$

On the one hand,

$$h^{\prime }(\theta )=⟨f^{\prime }((1-\theta )u+\theta v)(v-u),\psi ⟩.$$

On the other hand,

$$h(1)-h(0)=⟨f(v),\psi ⟩-⟨f(u),\psi ⟩=⟨f(v)-f(u),\psi ⟩=\parallel f(v)-f(u)\parallel .$$

Therefore

$\parallel f(v)-f(u)\parallel$ $=$ $|⟨f^{\prime }((1-\theta )u+\theta v)(v-u),\psi ⟩|$

$\le$ $\parallel \psi \parallel \parallel f^{\prime }((1-\theta )u+\theta v)(v-u)\parallel$

$=$ $\parallel f^{\prime }((1-\theta )u+\theta v)(v-u)\parallel$

$\le$ $\parallel f^{\prime }((1-\theta )u+\theta v)\parallel \parallel v-u\parallel$

$\le$ $\underset{w\in [u,v]}{sup}\parallel f^{\prime }(w)\parallel \parallel v-u\parallel .$

∎

7 Antiderivatives

Suppose that $X$ is a Banach space and that $f:[a,b]\to X$ be continuous. Define $F:[a,b]\to X$ by

$$F(x)={\int }_a^{x}f.$$

Let $x_0\in (a,b)$. For $x\in (a,b)$, we have

$$F(x)-F(x_0)={\int }_a^{x}f-{\int }_a^{x_0}f={\int }_{x_0}^{x}f=f(x_0)(x-x_0)+{\int }_{x_0}^x(f-f(x_0)),$$

from which it follows that $F$ is Fréchet differentiable at $x_0$, and that

$$DF(x_0)(x-x_0)=f(x_0)(x-x_0).$$

If we identify $f(x_0)\in X$ with the map $x\mapsto f(x_0)x$, namely if we say that $X=ℬ(ℝ,X)$, then $DF(x_0)=f(x_0)$.

Let $X$ be a normed space, let $Y$ be a Banach space, let $U$ be an open subset of $X$, and let $f\in C^1(U,Y)$. Suppose that $u,v\in U$ satisfy $[u,v]\subset U$. Write $I=(0,1)$ and define $\gamma :I\to U$ by $\gamma (t)=(1-t)u+tv$. We have

$$D\gamma (t)=v-u,t\in I,$$

and thus by Theorem 6,

$$D(f\circ \gamma )(t)=Df(\gamma (t))\circ D\gamma (t),t\in I,$$

that is,

$$D(f\circ \gamma )(t)=Df(\gamma (t))\circ (v-u),t\in I,$$

i.e.

$$D(f\circ \gamma )(t)=Df(\gamma (t))(v-u),t\in I.$$

If $t\in I$ and $t+h\in I$, then

$D(f\circ \gamma )(t+h)-D(f\circ \gamma )(t)$ $=$ $Df(\gamma (t+h))(v-u)-Df(\gamma (t))(v-u)$

$=$ $\left(Df(\gamma (t+h))-Df(\gamma (t))\right)(v-u),$

and hence

$$\parallel D(f\circ \gamma )(t+h)-D(f\circ \gamma )(t)\parallel \le \parallel Df(\gamma (t+h))-Df(\gamma (t))\parallel \parallel v-u\parallel .$$

Because $Df:U\to ℬ(X,Y)$ is continuous, it follows that

$$\parallel D(f\circ \gamma )(t+h)-D(f\circ \gamma )(t)\parallel \to 0$$

as $h\to 0$, i.e. that $D(f\circ \gamma )$ is continuous at $t$, and thus that

$$D(f\circ \gamma ):I\to ℬ(ℝ,Y)$$

is continuous. If we identify $ℬ(ℝ,Y)$ with $Y$, then

$$D(f\circ \gamma ):I\to Y.$$

On the one hand,

$${\int }_0^{1}D(f\circ \gamma )=(f\circ \gamma )(1)-(f\circ \gamma )(0)=f(v)-f(u).$$

On the other hand,

$${\int }_0^{1}D(f\circ \gamma )={\int }_0^{1}Df(\gamma (t))(v-u)𝑑t=\left({\int }_0^{1}Df((1-t)u+tv)𝑑t\right)(v-u);$$

here,

$${\int }_0^{1}Df((1-t)u+tv)𝑑t\in ℬ(X,Y).$$

Therefore

$$f(v)-f(u)=\left({\int }_0^{1}Df((1-t)u+tv)𝑑t\right)(v-u).$$