The Glivenko-Cantelli theorem

1 Narrow topology

Let $X$ be a metrizable space and let $C_b(X)$ be the Banach space of bounded continuous functions $X\to ℝ$, with the norm $\parallel f\parallel ={sup}_{x\in X}|f(x)|$. If $X$ is metrizable with the metric $d$, let $U_d(X)$ be the collection of bounded $d$-uniformly continuous functions $X\to ℝ$. This is a vector space and is a closed subset of $C_b(X)$, thus is itself a Banach space.

Let $X$ be a metrizable space and denote by $𝒫(X)$ the collection of Borel probability measures on $X$. The narrow topology on $\mathrm{P}\mathbf{}\mathrm{(}X\mathrm{)}$ is the coarsest topology on $𝒫(X)$ such that for every $f\in C_b(X)$, the mapping $\mu \mapsto {\int }_{X}f𝑑\mu$ is continuous $𝒫(X)\to ℝ$. It can be proved that if $X$ is metrizable with a metric $d$ and $D$ is a dense subset of $U_d(X)$, then the narrow topology is equal to the coarsest topology such that for each $f\in U_d(X)$, the mapping $\mu \mapsto {\int }_{X}f𝑑\mu$ is continuous $𝒫(X)\to ℝ$.

If $X$ is a separable metrizable space, then it is metrizable by a metric $d$ such that the metric space $(X,d)$ is totally bounded. It is a fact that if $(X,d)$ is a totally bounded metric space, then $U_d(X)$ is separable.

Theorem 1.

If $X$ is a separable metrizable space, then $X$ is metrizable by a metric $d$ for which there is a countable dense subset $D$ of $U_d\mathit{}\mathrm{(}X\mathrm{)}$ such that ${\mu }_n$ converges narrowly to $\mu$ if and only if

$${\int }_{X}f𝑑{\mu }_n\to {\int }_{X}f𝑑\mu ,f\in D.$$

2 Independent and identically distributed random variables

Let $(\mathrm{\Omega },S,P)$ be a probability space and let $X$ be a separable metric space, with the Borel $\sigma$-algebra ${ℬ}_X$. We say that a finite collection measurable functions ${\xi }_i:\mathrm{\Omega }\to X$, $1\le i\le n$, is independent if

$$P\left(\bigcap _{i=1}^n{\xi }_i^{-1}(A_i)\right)=\prod _{i=1}^{n}P({\xi }_i^{-1}(A_i)),A_1,\mathrm{\dots },A_n\in {ℬ}_X,$$

i.e.

$$P({\xi }_1\in A_1,\mathrm{\dots },{\xi }_n\in A_n)=P({\xi }_1\in A_1)\mathrm{\cdots }P({\xi }_n\in A_n),A_1,\mathrm{\dots },A_n\in {ℬ}_X.$$

We say that a family of measurable functions is independent if every finite subset of it is independent.

We say that two measurable functions $f,g:\mathrm{\Omega }\to X$ are identically distributed if the pushforward $f_{*}P$ of $P$ by $f$ is equal to the pushforward $g_{*}P$ of $P$ by $g$, i.e. $P(f^{-1}(A))=P(g^{-1}(A))$ for every $A\in {ℬ}_X$. We say that a family of measurable functions is identically distributed if any two of them are identically distributed.

3 Strong law of large numbers

If $\zeta \in L^1(P)$, the expectation of $\zeta$ is

$$E(\zeta )={\int }_{\mathrm{\Omega }}\zeta 𝑑P,$$

and by the change of variables theorem,

$${\int }_{\mathrm{\Omega }}\zeta (\omega )𝑑P(\omega )={\int }_{ℝ}xd({\zeta }_{*}P)(x).$$

The strong law of large numbers states that if ${\zeta }_1,{\zeta }_2,\mathrm{\dots }\in L^1(P)$ are independent and identically distributed, with common expectation $E_0$, then

$$P\left(\{\omega \in \mathrm{\Omega }:\sum _{i=1}^n\frac{{\zeta }_i(\omega )}{n}\to E_0\}\right)=1.$$

4 Sample distributions

Let $X$ be a separable metrizable space and let ${\xi }_1,{\xi }_2,\mathrm{\dots }$ be independent and identically distributed measurable functions $\mathrm{\Omega }\to X$. For $\omega \in \mathrm{\Omega }$, define ${\mu }_n^{\omega }$ on ${ℬ}_X$ by

$${\mu }_n^{\omega }=\sum _{i=1}^n\frac{1}{n}{\delta }_{{\xi }_i(\omega )},$$

which is a probability measure. We call the sequence ${\mu }_n^{\omega }$ the sample distribution of $\omega$.

The following is the Glivenko-Cantelli theorem, which shows that the sample distributions of a sequence of independent and identically distributed measurable functions converge narrowly almost everywhere to the common pushforward measure.

Theorem 2 (Glivenko-Cantelli theorem).

Let $\mathrm{(}\mathrm{\Omega }\mathrm{,}S\mathrm{,}P\mathrm{)}$ be a probability space, let $X$ be a separable metrizable space and let ${\xi }_{\mathrm{1}}\mathrm{,}{\xi }_{\mathrm{2}}\mathrm{,}\mathrm{\dots }$ be independent and identically distributed measurable functions $\mathrm{\Omega }\mathrm{\to }X$, with common pushforward measure $\mu$. Then

$$P\left(\{\omega \in \mathrm{\Omega }:{\mu }_n^{\omega }\to \mu \text{ narrowly}\}\right)=1.$$

Proof.

For $g\in C_b(X)$, $g\circ {\xi }_i:\mathrm{\Omega }\to ℝ$ is measurable bounded, hence belongs to $L^1(P)$. Also, ${(g\circ {\xi }_i)}_{*}P=g_{*}\mu$, so the sequence $g\circ {\xi }_i$ are identically distributed. We now check that the sequence is independent. Let $A_1,\mathrm{\dots },A_n\in {ℬ}_{ℝ}$. Then $g^{-1}(A_1),\mathrm{\dots },g^{-1}(A_n)\in {ℬ}_X$, and because ${\xi }_1,{\xi }_2,\mathrm{\dots }$ are independent,

$$P\left(\bigcap _{i=1}^n{\xi }_i^{-1}(g^{-1}(A_i))\right)=\prod _{i=1}^{n}P({\xi }_i^{-1}(g^{-1}(A_i))),$$

i.e.,

$$P\left(\bigcap _{i=1}^n{(g\circ {\xi }_i)}^{-1}(A_i)\right)=\prod _{i=1}^{n}P({(g\circ {\xi }_i)}^{-1}(A_i)),$$

showing that $(g\circ {\xi }_1),(g\circ {\xi }_2),\mathrm{\dots }$ are independent. For any $i$, by the change of variables theorem

$$E(g\circ {\xi }_i)={\int }_{\mathrm{\Omega }}g\circ {\xi }_i𝑑P={\int }_{X}gd({({\xi }_i)}_{*}P)={\int }_{X}g𝑑\mu ,$$

so the strong law of large numbers tells us that there is a set $N_g\in S$ with $P(N_g)=0$ such that for all $\omega \in \mathrm{\Omega }\setminus N_g$,

$$\sum _{i=1}^n\frac{(g\circ {\xi }_i)(\omega )}{n}\to {\int }_{X}g𝑑\mu .$$

But

$$\sum _{i=1}^n\frac{(g\circ {\xi }_i)(\omega )}{n}=\sum _{i=1}^n\frac{1}{n}{\int }_{X}g𝑑{\delta }_{{\xi }_i(\omega )}={\int }_{X}g𝑑{\mu }_n^{\omega },$$

so for all $\omega \in \mathrm{\Omega }\setminus N_g$,

$${\int }_{X}g𝑑{\mu }_n^{\omega }\to {\int }_{X}g𝑑\mu .$$

Because $X$ is separable, Theorem 1 tells us that there is a metric $d$ that induces the topology of $X$ and some countable dense subset $G$ of $U_d(X)$ such that a sequence ${\nu }_n$ in $𝒫(X)$ converges narrowly to $\nu$ if and only if

$${\int }_{X}g𝑑{\nu }_n\to {\int }_{X}g𝑑\nu ,g\in G.$$

Now let $N={\bigcup }_{g\in G}{N}_g$, which satisfies $P(N)=0$, and if $\omega \in \mathrm{\Omega }\setminus N$ then for each $g\in G$,

$${\int }_{X}g𝑑{\mu }_n^{\omega }\to {\int }_{X}g𝑑\mu .$$

This implies that for all ${\mu }_n^{\omega }$ converges narrowly to $\mu$. That is, there is a set $N\in S$ with $P(N)=0$ such that for all $\omega \in \mathrm{\Omega }\setminus N$, the sample distribution ${\mu }_n^{\omega }$ converges narrowly to the common pushforward measure $\mu$, proving the claim. ∎