The weak and strong laws of large numbers

1 Introduction

Using Egorov’s theorem, one proves that if a sequence of random variables $X_n$ converges almost surely to a random variable $X$ then $X_n$ converges in probability to $X$.

Let $X_n$ be a sequence of $L^1$ random variables, $n\ge 1$. A weak law of large numbers is a statement that

$$\frac{1}{n}\sum _{k=1}^n(X_k-E(X_k))$$ (1)

converges in probability to $0$. A strong law of large numbers is a statement that (1) converges almost surely to $0$. Thus, if the hypotheses assumed on the sequence of random variables are the same, a strong law implies a weak law.

We shall prove the weak law of large numbers for a sequence of independent identically distributed $L^1$ random variables, and the strong law of large for the same hypotheses. We give separate proofs for these theorems as an occasion to inspect different machinery, although to establish the weak law it thus suffices to prove the strong law. One reason to distinguish these laws is for cases when we impose different hypotheses.

We also prove Markov’s weak law of large numbers, which states that if $X_n$ is a sequence of $L^2$ random variables that are pairwise uncorrelated and

$$\frac{1}{n^2}\sum _{n=k}^n\mathrm{Var}(X_k)\to 0,$$

then $\frac{1}{n}{\sum }_{k=1}^n(X_k-E(X_k))$ converges to $0$ in $L^2$, from which it follows using Chebyshev’s inequality that it converges in probability to $0$. (We remark that a sequence of $L^2$ random variables converging in $L^2$ to $0$ does not imply that it converges almost surely to $0$, although there is indeed a subsequence that converges almost surely to $0$.)

If $(\mathrm{\Omega },ℱ,P)$ is a probability space, $(Y,𝒜)$ is a measurable space, and $T:(\mathrm{\Omega },ℱ)\to (Y,𝒜)$ is measurable, the pushforward measure of $P$ by $T$ is

$$(T_{*}P)(A)=P(T^{-1}(A)),A\in 𝒜.$$

Then $(Y,𝒜,T_{*}P)$ is a probability space. We remind ourselves of the change of variables theorem, which we shall use. Let $f:Y\to ℝ$ be a function. On the one hand, if $f\in L^1(T_{*}P)$ then $f\circ T\in L^1(P)$ and

$${\int }_{Y}fd(T_{*}P)={\int }_{\mathrm{\Omega }}f\circ T𝑑P.$$ (2)

On the other hand, if $f$ is $T_{*}P$-measurable and $f\circ T\in L^1(P)$, then $f\in L^1(T_{*}P)$ and

$${\int }_{Y}fd(T_{*}P)={\int }_{\mathrm{\Omega }}f\circ T𝑑P.$$

2 The weak law of large numbers

Suppose that $X_n:(\mathrm{\Omega },ℱ,P)\to ℝ$, $n\ge 1$, are independent identically distributed $L^1$ random variables, and write

$$S_n=\sum _{k=1}^n{X}_k,$$

for which $E(S_n)={\sum }_{k=1}^{n}E(X_k)=nE(X_1)$.

Lemma 1.

If $X_n$ are $L^{\mathrm{2}}$, then for any $\lambda \mathrm{>}\mathrm{0}$ and for any $n\mathrm{\ge }\mathrm{1}$,

$$P\left(\right|\frac{S_n}{n}-E(X_1)|\ge \lambda )\le \frac{E({|X_1-E(X_1)|}^2)}{n{\lambda }^2}.$$

Proof.

Using

$$S_n^2=\sum _{k=1}^n{X}_k^2+\sum _{j\ne k}{X}_j{X}_k,$$

we have

$E\left({\left|{\displaystyle \frac{S_n}{n}}-E(X_1)\right|}^2\right)$ $=E\left({\displaystyle \sum _{k=1}^n}{\displaystyle \frac{X_k^2}{n^2}}+{\displaystyle \sum _{j\ne k}}{\displaystyle \frac{X_j{X}_k}{n^2}}-2E(X_1){\displaystyle \frac{S_n}{n}}+E{(X_1)}^2\right)$

$={\displaystyle \sum _{k=1}^n}{\displaystyle \frac{E(X_k^2)}{n^2}}+{\displaystyle \sum _{j\ne k}}{\displaystyle \frac{E(X_j{X}_k)}{n^2}}-2E{(X_1)}^2+E{(X_1)}^2$

$={\displaystyle \sum _{k=1}^n}{\displaystyle \frac{E(X_1^2)}{n^2}}+{\displaystyle \sum _{j\ne k}}{\displaystyle \frac{E(X_j)E(X_k)}{n^2}}-E{(X_1)}^2$

$={\displaystyle \sum _{k=1}^n}{\displaystyle \frac{E(X_1^2)}{n^2}}+{\displaystyle \sum _{j\ne k}}{\displaystyle \frac{E{(X_1)}^2}{n^2}}+E{(X_1)}^2$

$={\displaystyle \frac{E(X_1^2)}{n}}+{\displaystyle \frac{E{(X_1)}^2}{n}}.$

On the other hand,

$$E({|X_1-E(X_1)|}^2)=E(X_1^2-2E(X_1)X_1+E{(X_1)}^2)=E(X_1^2)+E{(X_1)}^2.$$

So

$$E\left({\left|\frac{S_n}{n}-E(X_1)\right|}^2\right)=\frac{1}{n}E({|X_1-E(X_1)|}^2).$$

Using this and Chebyshev’s inequality,

$P\left(\right|{\displaystyle \frac{S_n}{n}}-E(X_1)|\ge \lambda )$ $\le {\displaystyle \frac{1}{{\lambda }^2}}E\left({\left|{\displaystyle \frac{S_n}{n}}-E(X_1)\right|}^2\right)$

$={\displaystyle \frac{1}{n{\lambda }^2}}E({|X_1-E(X_1)|}^2),$

proving the claim. ∎

We now prove the weak law of large numbers, which states that if $X_n$ are independent identically distributed $L^1$ random variables each with mean $0$, then $\frac{S_n}{n}$ converges in probability to $E(X_1)$. $Z_n=X_n-E(X_n)=X_n-E(X_1)$ are independent and identically distributed $L^1$ random variables with mean $0$, and if $Z_n$ converges in probability to $0$ then $X_n=Z_n+E(X_1)$ converges in probability to $E(X_1)$, showing that it suffices to prove the theorem when $E(X_1)=0$.

Theorem 2 (Weak law of large numbers).

Suppose that $E\mathit{}\mathrm{(}{X}_{\mathrm{1}}\mathrm{)}\mathrm{=}\mathrm{0}$. For each $ϵ\mathrm{>}\mathrm{0}$,

$$P\left(\right|\frac{S_n}{n}|\ge ϵ)\to 0,n\to \mathrm{\infty }.$$

Proof.

For $n\ge 1$ and $1\le k\le n$, let

$$Y_{n,k}=1_{\{|X_k|\le n{ϵ}^3\}}{X}_k=f\circ X_k,$$

where $f(x)=1_{[-n{ϵ}^3,n{ϵ}^3]}(x)\cdot x$. Because $X_1,\mathrm{\dots },X_k$ are independent and identically distributed, so are $Y_{n,1},\mathrm{\dots },Y_{n,n}$. Moreover, $E(Y_{n,k}^2)\le {(n{ϵ}^3)}^2$, so each $Y_{n,k}$ belongs to $L^2$. Let

$$T_n=\sum _{k=1}^n{Y}_{n,k},$$

for which $E(T_n)=nE(Y_{n,1})$.

If $\omega \in {\bigcap }_{k=1}^n\{|X_k|\le n{ϵ}^3\}$, then

$$T_n(\omega )=\sum _{k=1}^n{Y}_{n,k}(\omega )=\sum _{k=1}^n{X}_k(\omega )=S_n(\omega ),$$

and using this and Lemma 1, for $t>0$ we have

$P(|S_n-E(T_n)|\ge t)$ $=P(\{|S_n-E(T_n)|\ge t\}\cap {\displaystyle \bigcap _{k=1}^n}\{|X_k|\le n{ϵ}^3\})$

$+P(\{|S_n-E(T_n)|\ge t\}\cap {\displaystyle \bigcup _{k=1}^n}\{|X_k|>n{ϵ}^3\})$

$\le P(|T_n-E(T_n)|\ge t)+P\left({\displaystyle \bigcup _{k=1}^n}\{|X_k|>n{ϵ}^3\}\right)$

$=P\left(\right|{\displaystyle \frac{T_n}{n}}-E(Y_{n,1})|\ge {\displaystyle \frac{t}{n}})+P\left({\displaystyle \bigcup _{k=1}^n}\{|X_k|>n{ϵ}^3\}\right)$

$\le {\displaystyle \frac{E({|Y_{n,1}-E(Y_{n,1})|}^2)}{n\cdot {\left(\frac{t}{n}\right)}^2}}+{\displaystyle \sum _{k=1}^n}P(|X_k|>n{ϵ}^3)$

$={\displaystyle \frac{n}{t^2}}\cdot (E(Y_{n,1}^2)-E{(Y_{n,1})}^2)+nP(|X_1|>n{ϵ}^3)$

$\le {\displaystyle \frac{n}{t^2}}\cdot E(Y_{n,1}^2)+nP(|X_1|>n{ϵ}^3).$

For $t=nϵ$ this is

$P(|S_n-E(T_n)|\ge nϵ)$ $\le {\displaystyle \frac{1}{n{ϵ}^2}}\cdot E(Y_{n,1}^2)+nP(|X_1|>n{ϵ}^3)$

$={\displaystyle \frac{1}{n{ϵ}^2}}\cdot E(1_{\{|X_1|\le n{ϵ}^3\}}{X}_1^2)+nP(|X_1|>n{ϵ}^3)$

$\le {\displaystyle \frac{1}{n{ϵ}^2}}\cdot E(1_{\{|X_1|\le n{ϵ}^3\}}n{ϵ}^3|X_1|)+nP(|X_1|>n{ϵ}^3)$

$\le ϵE(|X_1|)+nP(|X_1|>n{ϵ}^3).$

Now,

$$n{ϵ}^{3}P(|X_1|>n{ϵ}^3)=n{ϵ}^3{\int }_{(n{ϵ}^3,\mathrm{\infty })}d(X_1{}_{*}P)(y)\le {\int }_{(n{ϵ}^3,\mathrm{\infty })}yd(X_1{}_{*}P)(y),$$

which tends to $0$ as $n\to \mathrm{\infty }$ because

Therefore,

$$\underset{n}{lim\; sup}P(|S_n-E(T_n)|\ge nϵ)\le ϵE(|X_1|),$$

that is,

$$\underset{n}{lim\; sup}P\left(\right|\frac{S_n-E(T_n)}{n}|\ge ϵ)\le ϵE(|X_1|),$$

which implies that $\frac{S_n-E(T_n)}{n}$ converges in probability to $0$.

Because $E(X_1)=0$,

$$E(1_{\{|X_1|\le n{ϵ}^3\}}{X}_1)+E(1_{\{|X_1|>n{ϵ}^3\}}{X}_1)=0,$$

and hence

$$|E(T_n)|=|nE(Y_{n,1})|=n|E(1_{\{|X_1|\le n{ϵ}^3\}}{X}_1)|=n|E(1_{\{|X_1|>n{ϵ}^3\}}{X}_1)|,$$

thus

$$\frac{|E(T_n)|}{n}=|E(1_{\{|X_1|>n{ϵ}^3\}}{X}_1)|\le E(1_{\{|X_1|>n{ϵ}^3\}}|X_1|),$$

which tends to $0$ as $n\to \mathrm{\infty }$, because . Thus $\frac{E(T_n)}{n}$ converges in probability to $0$, and therefore $\frac{S_n}{n}$ converges in probability to $0$, completing the proof. ∎

Lemma 3 (Bienaymé’s formula).

If $X_n$, $n\mathrm{\ge }\mathrm{1}$, are $L^{\mathrm{2}}$ random variables that are pairwise uncorrelated, then

$$\mathrm{Var}\left(\sum _{k=1}^n{X}_k\right)=\sum _{k=1}^n\mathrm{Var}(X_k).$$

Proof.

Let $Y_k=X_k-E(X_k)$. Using that the $X_k$ are pairwise uncorrelated, for $j\ne k$ we have

$E(Y_j{Y}_k)$ $=E(X_j{X}_k-X_{j}E(X_k)-X_{k}E(X_j)+E(X_j)E(X_k))$

$=E(X_j)E(X_k)-E(X_j)E(X_k)-E(X_k)E(X_j)+E(X_j)E(X_k)$

$=0,$

showing that the $Y_k$ are pairwise uncorrelated. Then, because $E(S_n)={\sum }_{k=1}^{n}E(X_k)$,

$\mathrm{Var}(S_n)$ $=E\left({\left(S_n-{\displaystyle \sum _{k=1}^n}E(X_k)\right)}^2\right)$

$=E\left({\left({\displaystyle \sum _{k=1}^n}{Y}_k\right)}^2\right)$

$=E\left({\displaystyle \sum _{k=1}^n}{Y}_k^2+{\displaystyle \sum _{j\ne k}}{Y}_j{Y}_k\right)$

$={\displaystyle \sum _{k=1}^n}E(Y_k^2)+{\displaystyle \sum _{j\ne k}}E(Y_j)E(Y_k)$

$={\displaystyle \sum _{k=1}^n}E(Y_k^2),$

and as $E(Y_k^2)=\mathrm{Var}(X_k)$, we have

$$\mathrm{Var}(S_n)=\sum _{k=1}^n\mathrm{Var}(X_k).$$

∎

We now prove a weak law of large numbers, that is sometimes attributed to Markov, that neither supposes that the random variables are independent nor supposes that they are identically distributed. We remind ourselves that if a sequence $Y_n$ of $L^2$ random variables converges to $0$ in $L^2$ then it converges in probability to $0$; this is proved using Chebyshev’s inequality.

Theorem 4 (Markov’s weak law of large numbers).

If $X_n$, $n\mathrm{\ge }\mathrm{1}$, are $L^{\mathrm{2}}$ random variables that are pairwise uncorrelated and which satisfy

$$\frac{1}{n^2}\sum _{n=k}^n\mathrm{Var}(X_k)\to 0,$$

then $\frac{\mathrm{1}}{n}\mathit{}{\mathrm{\sum }}_{k\mathrm{=}\mathrm{1}}^n\mathrm{(}{X}_k\mathrm{-}E\mathit{}\mathrm{(}{X}_k\mathrm{)}\mathrm{)}$ converges to $\mathrm{0}$ in $L^{\mathrm{2}}$ and thus in probability.

Proof.

Because $E\left({\sum }_{k=1}^n{X}_k\right)={\sum }_{k=1}^{n}E(X_k)$ and $\mathrm{Var}(X)=E({(X-E(X))}^2)$, using Bienaymé’s formula we get

$E\left({\left({\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^n}(X_k-E(X_k))\right)}^2\right)$ $={\displaystyle \frac{1}{n^2}}E\left({\left({\displaystyle \sum _{k=1}^n}{X}_k-{\displaystyle \sum _{k=1}^n}E(X_k)\right)}^2\right)$

$={\displaystyle \frac{1}{n^2}}\mathrm{Var}\left({\displaystyle \sum _{k=1}^n}{X}_k\right)$

$={\displaystyle \frac{1}{n^2}}{\displaystyle \sum _{k=1}^n}\mathrm{Var}(X_k).$

Thus

$$E\left({\left(\frac{1}{n}\sum _{k=1}^n(X_k-E(X_k))\right)}^2\right)=\frac{1}{n^2}\sum _{k=1}^n\mathrm{Var}(X_k)\to 0$$

as $n\to \mathrm{\infty }$, namely, $\frac{1}{n}{\sum }_{k=1}^n(X_k-E(X_k))$ converges to $0$ in $L^2$ as $n\to \mathrm{\infty }$, proving the claim. ∎

3 Ergodic theory

We here assemble machinery and results that we will use to prove the strong law of large numbers. For a probability space $(\mathrm{\Omega },ℱ,P)$, a function $T:\mathrm{\Omega }\to \mathrm{\Omega }$ is said to be a measure-preserving transformation if (i) it is measurable, and (ii) $T_{*}P=P$. To say that $T_{*}P=P$ means that for any $A\in ℱ$, $(T_{*}P)(A)=P(A)$, i.e. $P(T^{-1}(A))=P(A)$.

A collection $𝒜$ of subsets of $\mathrm{\Omega }$ is called an algebra of sets if (i) $\mathrm{\varnothing }\in 𝒜$, (ii) $\mathrm{\Omega }\in 𝒜$, (iii) if $A,B\in 𝒜$ then $A\cup B,A\setminus B\in 𝒜$. If $𝒢$ is a nonempty collection of subsets of $\mathrm{\Omega }$, it is a fact that there is a unique algebra of sets $A(𝒢)$ that (i) contains $𝒢$ and (ii) is contained in any algebra of sets that contains $𝒢$. We call $A(𝒢)$ the algebra of sets generated by $\mathrm{G}$.

A collection $𝒮$ of subsets of $\mathrm{\Omega }$ is called a semialgebra of sets if (i) $\mathrm{\varnothing }\in 𝒮$, (ii) $\mathrm{\Omega }\in 𝒮$, (iii) if $A,B\in 𝒮$ then $A\cap B\in 𝒮$, (iv) if $A,B\in 𝒮$ then there are pairwise disjoint $E_1,\mathrm{\dots },E_n\in 𝒮$ such that

$$A\setminus B=\bigcup _{k=1}^n{E}_k.$$

It is a fact that $A(𝒮)$ is equal to the collection of all unions of finitely many pairwise disjoint elements of $𝒮$.

A nonempty collection $ℳ$ of subsets of $\mathrm{\Omega }$ is called a monotone class if whenever $A_n\in ℳ$, $A_n\subset A_{n+1}$ (an increasing sequence of sets), implies that ${\bigcup }_n{A}_n\in ℳ$ and $A_n\in ℳ$, $A_{n+1}\subset A_n$ (a decreasing sequence of sets), implies that ${\bigcap }_n{A}_n\in ℳ$. In other words, a monotone class is a nonempty collection of subsets of $\mathrm{\Omega }$ such that if $A_n$ is a monotone sequence in $ℳ$ then ${lim}_{n\to \mathrm{\infty }}{A}_n\in ℳ$. If $𝒢$ is a nonempty collection of subsets of $\mathrm{\Omega }$, it is a fact that there is a unique monotone class $M(𝒢)$ that (i) contains $𝒢$ and (ii) is contained in any monotone class that contains $𝒢$. We call $M(𝒢)$ the monotone class generated by $\mathrm{G}$. The monotone class theorem states that if $𝒜$ is an algebra of sets then $\sigma (𝒜)=M(𝒜)$.

The following lemma gives conditions under which we can establish that a function is measure-preserving.

Lemma 5.

Let $T\mathrm{:}\mathrm{\Omega }\mathrm{\to }\mathrm{\Omega }$ be a function and suppose that $\mathrm{S}$ is a semialgebra of sets for which $\mathrm{F}$ is the $\sigma$-algebra generated by $\mathrm{S}$. If (i) $T^{\mathrm{-}\mathrm{1}}\mathit{}\mathrm{(}A\mathrm{)}\mathrm{\in }\mathrm{F}$ for each $A\mathrm{\in }\mathrm{S}$ and (ii) $P\mathit{}\mathrm{(}{T}^{\mathrm{-}\mathrm{1}}\mathit{}\mathrm{(}A\mathrm{)}\mathrm{)}\mathrm{=}P\mathit{}\mathrm{(}A\mathrm{)}$ for each $A\mathrm{\in }\mathrm{S}$, then $T$ is measure-preserving.

Proof.

Let

$$𝒞=\{A\in ℱ:T^{-1}(A)\in ℱ,P(T^{-1}(A))=P(A)\};$$

we wish to prove that $𝒞=ℱ$.

If $A_n\in 𝒞$ is an increasing sequence, let $A={\bigcup }_{n=1}^{\mathrm{\infty }}{A}_n$. Then, as $T^{-1}(A_n)\in ℱ$ for each $n$,

$$T^{-1}(A)=\bigcup _{n=1}^{\mathrm{\infty }}{T}^{-1}(A_n)\in ℱ$$

and as (i) $T^{-1}(A_n)$ is an increasing sequence, (ii) $P$ is continuous from below, and (iii) $P(T^{-1}(A_n))=P(A_n)$,

$P(T^{-1}(A))$ $=P\left({\displaystyle \bigcup _{n=1}^{\mathrm{\infty }}}{T}^{-1}(A_n)\right)$

$=\underset{n\to \mathrm{\infty }}{lim}P(T^{-1}(A_n))$

$=\underset{n\to \mathrm{\infty }}{lim}P(A_n)$

$=P\left({\displaystyle \bigcup _{n=1}^{\mathrm{\infty }}}{A}_n\right)$

$=P(A),$

and hence $A\in 𝒞$. If $A_n\in 𝒞$ is a decreasing sequence, let $A={\bigcap }_{n=1}^{\mathrm{\infty }}{A}_n$. Because $T^{-1}(A_n)\in ℱ$,

$$T^{-1}(A)=\bigcap _{n=1}^{\mathrm{\infty }}{T}^{-1}(A_n)\in ℱ,$$

and as (i) $T^{-1}(A_n)$ is a decreasing sequence, (ii) $P$ is continuous from above, and (iii) $P(T^{-1}(A_n))=P(A_n)$,

$P(T^{-1}(A))$ $=P\left({\displaystyle \bigcap _{n=1}^{\mathrm{\infty }}}{T}^{-1}(A_n)\right)$

$=\underset{n\to \mathrm{\infty }}{lim}P(T^{-1}(A_n))$

$=\underset{n\to \mathrm{\infty }}{lim}P(A_n)$

$=P\left({\displaystyle \bigcap _{n=1}^{\mathrm{\infty }}}{A}_n\right)$

$=P(A),$

and hence $A\in 𝒞$. Therefore, $𝒞$ is a monotone class.

$𝒮\subset 𝒞$. If $A\in A(𝒮)$, then there are pairwise disjoint $A_1,\mathrm{\dots },A_n\in 𝒮$ with $A={\bigcup }_{k=1}^n{A}_k$. As $T^{-1}(A_k)\in ℱ$,

$$T^{-1}(A)=\bigcup _{k=1}^n{T}^{-1}(A_k)\in ℱ.$$

As the $A_k$ are pairwise disjoint, so are the sets $T^{-1}(A_k)$, so, because $P(T^{-1}(A_k))=P(A_k)$,

$P(T^{-1}(A))$ $=P\left({\displaystyle \bigcup _{k=1}^n}{T}^{-1}(A_k)\right)$

$={\displaystyle \sum _{k=1}^n}P(T^{-1}(A_k))$

$={\displaystyle \sum _{k=1}^n}P(A_k)$

$=P\left({\displaystyle \bigcup _{k=1}^n}{A}_k\right)$

$=P(A).$

Therefore $A\in 𝒞$. This shows that $A(𝒮)\subset 𝒞$.

The monotone class theorem tells us $\sigma (A(𝒮))=M(A(𝒮))$. On the one hand, $ℱ=\sigma (𝒮)$ and hence $ℱ=\sigma (A(𝒮))$. On the other hand, $A(𝒮)\subset 𝒞$ and the fact that $𝒞$ is a monotone class yield

$$M(A(𝒮))\subset M(𝒞)=𝒞.$$

Therefore

$$ℱ\subset 𝒞.$$

Of course $𝒞\subset ℱ$, so $𝒞=ℱ$, proving the claim. ∎

Let $(\mathrm{\Omega },ℱ,P)$ be a probability space. A measure-preserving transformation $T:\mathrm{\Omega }\to \mathrm{\Omega }$ is called ergodic if $A\in ℱ$ and $T^{-1}(A)=A$ implies that $P(A)=0$ or $P(A)=1$. It is proved using the Birkhoff ergodic theorem that for a measure-preserving transformation $T:\mathrm{\Omega }\to \mathrm{\Omega }$, $T$ is ergodic if and only if for all $A,B\in ℱ$,

$$\frac{1}{n}\sum _{k=0}^{n-1}P(T^{-1}(A)\cap B)\to P(A)P(B).$$

A measure-preserving transformation $T:\mathrm{\Omega }\to \mathrm{\Omega }$ is called weak-mixing if for all $A,B\in ℱ$,

$$\frac{1}{n}\sum _{k=0}^{n-1}|P(T^{-k}(A)\cap B)-P(A)P(B)|\to 0.$$

It is immediate that a weak-mixing transformation is ergodic.

A measure-preserving transformation $T:\mathrm{\Omega }\to \mathrm{\Omega }$ is called strong-mixing if for all $A,B\in ℱ$,

$$P(T^{-n}(A)\cap B)\to P(A)P(B).$$

If a sequence of real numbers $a_n$ tends to $0$, then

$$\frac{1}{n}\sum _{k=0}^{n-1}|a_k|\to 0,$$

and using this we check that a strong-mixing transformation is weak-mixing.

The following statement gives conditions under which a measure-preserving transformation is ergodic, weak-mixing, or strong-mixing.

Theorem 6.

Let $\mathrm{(}\mathrm{\Omega }\mathrm{,}\mathrm{F}\mathrm{,}P\mathrm{)}$ be a probability space, let $\mathrm{S}$ be a semialgebra that generates $\mathrm{F}$, and let $T\mathrm{:}\mathrm{\Omega }\mathrm{\to }\mathrm{\Omega }$ be a measure-preserving transformation.

1. $T$ is ergodic if and only if for all $A,B\in 𝒮$,

   $$\frac{1}{n}\sum _{k=0}^{n-1}P(T^{-k}(A)\cap B)\to P(A)P(B).$$

2. $T$ is weak-mixing if and only if for all $A,B\in 𝒮$,

   $$\frac{1}{n}\sum _{k=0}^{n-1}|P(T^{-k}(A)\cap B)-P(A)P(B)|\to 0.$$

3. $T$ is strong-mixing if and only if for all $A,B\in 𝒮$,

   $$P(T^{-n}(A)\cap B)\to P(A)P(B).$$

4 The strong law of large numbers

Let $\mu$ be a Borel probability measure on $ℝ$ with finite first moment:

We shall specify when we use the hypothesis that $\mu$ has finite first moment; until we say so, what we say merely supposes that it is a Borel probability measure on $ℝ$.

For $n\ge 0$, let ${\mathrm{\Omega }}_n=ℝ$, let ${ℬ}_n={ℬ}_{ℝ}$, the Borel $\sigma$-algebra of $ℝ$, and let ${\mu }_n=\mu$, for which $({\mathrm{\Omega }}_n,{ℬ}_n,{\mu }_n)$ is a probability space. Let $\mathrm{\Omega }={\prod }_{n=0}^{\mathrm{\infty }}{\mathrm{\Omega }}_n$. A cylinder set is a subset of $\mathrm{\Omega }$ of the form

$$\prod _{n=0}^{\mathrm{\infty }}{A}_n,$$

where $A_n\in {ℬ}_n$ for each $n$ and where $\{n\ge 0:A_n\ne {\mathrm{\Omega }}_n\}$ is finite. We denote by $𝒞$ the collection of all cylinder sets. It is a fact that $𝒞$ is a semialgebra of sets.

The product $\sigma$-algebra is $ℱ=\sigma (𝒞)$, the $\sigma$-algebra generated by the collection of all cylinder sets. The product measure, which we denote by $P$, is the unique probability measure on $ℱ$ such that for any cylinder set ${\prod }_{n=0}^{\mathrm{\infty }}{A}_n$,

$$P(\prod _{n=0}^{\mathrm{\infty }}{A}_n)=\prod _{n=0}^{\mathrm{\infty }}{\mu }_n(A_n).$$

Let ${\pi }_n:\mathrm{\Omega }\to {\mathrm{\Omega }}_n$, $n\ge 0$, be the projection map. Define $\tau :\mathrm{\Omega }\to \mathrm{\Omega }$ by

$$\tau ({\omega }_0,{\omega }_1,\mathrm{\dots })=({\omega }_1,{\omega }_2,\mathrm{\dots }),$$

the left shift map. In other words, for $n\ge 0$, $\tau$ satisfies ${\pi }_n\circ \tau ={\pi }_{n+1}$, and so ${\pi }_n={\pi }_0\circ ({\tau }^n)$.

Lemma 7.

$\tau$ is measure-preserving.

Proof.

For $A={\prod }_{n=0}^{\mathrm{\infty }}{A}_n\in 𝒞$,

$${\tau }^{-1}(A)=\prod _{n=0}^{\mathrm{\infty }}{B}_n=B,$$

where $B_0={\mathrm{\Omega }}_0$ and for $n\ge 1$, $B_n=A_{n-1}$. $B$ is a cylinder set so a fortiori belongs to $ℱ$, and

$$P(B)=\prod _{n=0}^{\mathrm{\infty }}{\mu }_n(B_n)={\mu }_0({\mathrm{\Omega }}_0)\cdot \prod _{n=1}^{\mathrm{\infty }}{\mu }_n(B_n)=\prod _{n=1}^{\mathrm{\infty }}{\mu }_n(A_{n-1})=\prod _{n=1}^{\mathrm{\infty }}{\mu }_{n-1}(A_{n-1}),$$

so

$$P({\tau }^{-1}(A))=\prod _{n=0}^{\mathrm{\infty }}{\mu }_n(A_n)=P(A).$$

Therefore by Lemma 5, because $𝒞$ is a semialgebra that generates $ℱ$, it follows that $\tau$ is measure-preserving. ∎

Lemma 8.

$\tau$ is strong-mixing.

Proof.

Let $A={\prod }_{n=0}^{\mathrm{\infty }}{A}_n$ and $B={\prod }_{n=0}^{\mathrm{\infty }}{B}_n$ be cylinder sets. For $n\ge 0$,

$${\tau }^{-n}(A)=\prod _{m=0}^{\mathrm{\infty }}{C}_m,$$

where $C_m={\mathrm{\Omega }}_m$ for $0\le m\le n-1$ and $C_m=A_{m-n}$ for $m\ge n$. Because $A$ and $B$ are cylinder sets, there is some $N$ such that when $m\ge N$, $A_m={\mathrm{\Omega }}_m$ and $B_m={\mathrm{\Omega }}_m$. Thus for $n\ge N$,

${\tau }^{-n}(A)\cap B)$ $={\displaystyle \prod _{m=0}^{\mathrm{\infty }}}{C}_m\cap {\displaystyle \prod _{m=0}^{\mathrm{\infty }}}{B}_m$

$={\displaystyle \prod _{m=0}^{\mathrm{\infty }}}(C_m\cap B_m)$

$={\displaystyle \prod _{m=0}^{n-1}}(C_m\cap B_m)\times {\displaystyle \prod _{m=n}^{\mathrm{\infty }}}(C_m\cap B_m)$

$={\displaystyle \prod _{m=0}^{n-1}}({\mathrm{\Omega }}_m\cap B_m)\times {\displaystyle \prod _{m=n}^{\mathrm{\infty }}}(A_{m-n}\cap {\mathrm{\Omega }}_m)$

$={\displaystyle \prod _{m=0}^{n-1}}{B}_m\times {\displaystyle \prod _{m=n}^{\mathrm{\infty }}}{A}_{m-n}.$

Hence

$P({\tau }^{-n}(A)\cap B)$ $={\displaystyle \prod _{m=0}^{n-1}}{\mu }_m(B_m)\cdot {\displaystyle \prod _{m=n}^{\mathrm{\infty }}}{\mu }_m(A_{m-n})$

$={\displaystyle \prod _{m=0}^{n-1}}{\mu }_m(B_m)\cdot {\displaystyle \prod _{m=0}^{\mathrm{\infty }}}{\mu }_{m+n}(A_m)$

$={\displaystyle \prod _{m=0}^{n-1}}{\mu }_m(B_m)\cdot {\displaystyle \prod _{m=0}^{\mathrm{\infty }}}{\mu }_m(A_m)$

$=P(B)\cdot P(A).$

That is, there is some $N$ such that when $n\ge N$,

$$P({\tau }^{-n}(A)\cap B)=P(A)P(B),$$

and so a fortiori,

$$\underset{n\to \mathrm{\infty }}{lim}P({\tau }^{-n}(A)\cap B)=P(A)P(B).$$

Therefore, because the cylinder sets generate the $\sigma$-algebra $ℱ$, by Theorem 3 we get that $\tau$ is strong-mixing. ∎

We now use the hypothesis that $\mu$ has finite first moment.

Lemma 9.

${\pi }_0\in L^1(P)$.

Proof.

$T={\pi }_0:\mathrm{\Omega }\to {\mathrm{\Omega }}_0$ is measurable, and $T_{*}P={\mu }_0$. The statement that $\mu ={\mu }_0$ has finite first moment means that $f:{\mathrm{\Omega }}_0\to ℝ$ defined by $f(x)=|x|$ belongs to $L^1({\mu }_0)$. Therefore by the change of variables theorem (2), we have $f\circ T\in L^1(P)$. ∎

Lemma 10.

For almost all $\omega \mathrm{\in }\mathrm{\Omega }$,

$$\frac{1}{n}\sum _{k=0}^{n-1}{\pi }_k(\omega )\to {\int }_{ℝ}t𝑑\mu (t).$$

Proof.

Because $\tau :\mathrm{\Omega }\to \mathrm{\Omega }$ is ergodic and ${\pi }_0\in L^1(P)$, the Birkhoff ergodic theorem tells us that for almost all $\omega \in \mathrm{\Omega }$,

$$\frac{1}{n}\sum _{k=0}^{n-1}{\pi }_0({\tau }^k(\omega ))\to {\int }_{\mathrm{\Omega }}{\pi }_0(\omega )𝑑P(\omega ),$$

i.e.,

$$\frac{1}{n}\sum _{k=0}^{n-1}{\pi }_k(\omega )\to {\int }_{{\mathrm{\Omega }}_0}{\omega }_0𝑑{\mu }_0({\omega }_0),$$

proving the claim. ∎

We will use Lemma 10 to prove the strong law of large numbers. First we prove two lemmas about joint distributions.

Lemma 11.

If $X_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{X}_n\mathrm{:}\mathrm{\Omega }\mathrm{\to }\mathrm{R}$ and $Y_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{Y}_n\mathrm{:}{\mathrm{\Omega }}^{\mathrm{\prime }}\mathrm{\to }\mathrm{R}$ are random variables with the same joint distribution and for each $\mathrm{1}\mathrm{\le }k\mathrm{\le }n$, ${\mathrm{\Phi }}_k\mathrm{:}{\mathrm{R}}^n\mathrm{\to }\mathrm{R}$ is a continuous function, then for $U_k\mathrm{=}{\mathrm{\Phi }}_k\mathit{}\mathrm{(}{X}_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{X}_n\mathrm{)}$ and $V_k\mathrm{=}{\mathrm{\Phi }}_k\mathit{}\mathrm{(}{Y}_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{Y}_n\mathrm{)}$, the random variables $U_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{U}_n$ and $V_{\mathrm{1}}\mathrm{,}\mathrm{\dots }\mathrm{,}{V}_n$ have the same joint distribution.

Proof.

Write $X=(X_1,\mathrm{\dots },X_n)$, $Y=(Y_1,\mathrm{\dots },Y_n)$, $U=(U_1,\mathrm{\dots },U_n)$, and $V=(V_1,\mathrm{\dots },V_n)$, which are Borel measurable. Let $\mathrm{\Phi }=({\mathrm{\Phi }}_1,\mathrm{\dots },{\mathrm{\Phi }}_n)$, which is continuous ${ℝ}^n\to {ℝ}^n$, and for which

$$U=\mathrm{\Phi }(X),V=\mathrm{\Phi }(Y).$$

To show that $U_1,\mathrm{\dots },U_n$ and $V_1,\mathrm{\dots },V_n$ have the same joint distribution means to show that $U_{*}P=V_{*}{P}^{\prime }$. Let $A\in {ℬ}_{{ℝ}^n}$, for which ${\mathrm{\Phi }}^{-1}(A)\in {ℬ}_{{ℝ}^n}$ and

$(U_{*}P)(A)$ $=P(U^{-1}(A))$

$=P(X^{-1}({\mathrm{\Phi }}^{-1}(A)))$

$=P^{\prime }(Y^{-1}({\mathrm{\Phi }}^{-1}(A)))$

$=P^{\prime }(V^{-1}(A))$

$=(V_{*}{P}^{\prime })(A),$

where, because $X_1,\mathrm{\dots },X_n$ and $Y_1,\mathrm{\dots },Y_n$ have the same joint distribution,

$$P(X^{-1}({\mathrm{\Phi }}^{-1}(A)))=(X_{*}P)({\mathrm{\Phi }}^{-1}(A))=(Y_{*}{P}^{\prime })({\mathrm{\Phi }}^{-1}(A))=P^{\prime }(Y^{-1}({\mathrm{\Phi }}^{-1}(A))).$$

∎

Lemma 12.

If sequences of random variables $X_n\mathrm{:}\mathrm{(}\mathrm{\Omega }\mathrm{,}\mathrm{F}\mathrm{,}P\mathrm{)}\mathrm{\to }\mathrm{R}$ and $Y_n\mathrm{:}\mathrm{(}{\mathrm{\Omega }}^{\mathrm{\prime }}\mathrm{,}{\mathrm{F}}^{\mathrm{\prime }}\mathrm{,}{P}^{\mathrm{\prime }}\mathrm{)}\mathrm{\to }\mathrm{R}$, $n\mathrm{\ge }\mathrm{1}$, have the same finite-dimensional distributions and there is some $a\mathrm{\in }\mathrm{R}$ such that $X_n$ converges to $a$ almost surely, then $Y_n$ converges to $a$ almost surely.

Proof.

That the sequences $X_n$ and $Y_n$ have the same finite-dimensional distributions means that for each $n\ge 1$,

$${(X_1,\mathrm{\dots },X_n)}_{*}P={(Y_1,\mathrm{\dots },Y_n)}_{*}{P}^{\prime },$$

i.e., for each $n\ge 1$ and for each $A\in {ℬ}_{{ℝ}^n}$,

$$P((X_1,\mathrm{\dots },X_n)\in A)=P^{\prime }((Y_1,\mathrm{\dots },Y_n)\in A).$$

Define

$E_{k,N,n}$ $=\{\omega \in \mathrm{\Omega }:|X_n(\omega )-a|\le {\displaystyle \frac{1}{k}}\}$

$F_{k,N,n}$ $=\{\omega \in {\mathrm{\Omega }}^{\prime }:|Y_n(\omega )-a|\le {\displaystyle \frac{1}{k}}\}.$

Then define

$E$ $={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{\displaystyle \bigcup _{N=1}^{\mathrm{\infty }}}{\displaystyle \bigcap _{n=N}^{\mathrm{\infty }}}{E}_{k,N,n}={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{\displaystyle \bigcup _{N=1}^{\mathrm{\infty }}}{E}_{k,N}={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{E}_k$

$F$ $={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{\displaystyle \bigcup _{N=1}^{\mathrm{\infty }}}{\displaystyle \bigcap _{n=N}^{\mathrm{\infty }}}{F}_{k,N,n}={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{\displaystyle \bigcup _{N=1}^{\mathrm{\infty }}}{F}_{k,N}={\displaystyle \bigcap _{k=1}^{\mathrm{\infty }}}{F}_k,$

and

$G_{k,N,n}$ $={\displaystyle \bigcap _{m=N}^n}{E}_{k,N,m}$

$H_{k,N,n}$ $={\displaystyle \bigcap _{m=N}^n}{F}_{k,N,m}.$

Because $(X_1,\mathrm{\dots },X_n)$ and $(Y_1,\mathrm{\dots },Y_n)$ have the same distribution,

$$P(G_{k,N,n})=P^{\prime }(H_{k,N,n}).$$

But

$$E_{k,N}=\bigcap _{n=N}^{\mathrm{\infty }}{E}_{k,N,n}=\bigcap _{n=N}^{\mathrm{\infty }}{G}_{k,N,n}=\underset{n\to \mathrm{\infty }}{lim}{G}_{k,N,n}$$

and

$$F_{k,N}=\underset{n\to \mathrm{\infty }}{lim}{H}_{k,N,n},$$

so

$$P(E_{k,N})=P^{\prime }(F_{k,N}).$$

Then

$$P(E_k)=\underset{N\to \mathrm{\infty }}{lim}P(E_{k,N})=\underset{N\to \mathrm{\infty }}{lim}{P}^{\prime }(F_{k,N})=P^{\prime }(F_k).$$

Then

$$P(E)=\underset{k\to \mathrm{\infty }}{lim}P(E_k)=\underset{k\to \mathrm{\infty }}{lim}{P}^{\prime }(F_k)=P^{\prime }(F).$$

That the sequence $X_n$ converges almost surely means that $P(E)=1$, and therefore $P^{\prime }(F)=1$, i.e. the sequence $Y_n$ converges almost surely. ∎

We now use what we have established to prove the strong law of large numbers. (We write $({\mathrm{\Omega }}^{\prime },{ℱ}^{\prime },P^{\prime })$ for a probability space because $(\mathrm{\Omega },ℱ,P)$ denotes the product probability space constructed already.)

Theorem 13.

If $X_n\mathrm{:}\mathrm{(}{\mathrm{\Omega }}^{\mathrm{\prime }}\mathrm{,}{\mathrm{F}}^{\mathrm{\prime }}\mathrm{,}{P}^{\mathrm{\prime }}\mathrm{)}\mathrm{\to }\mathrm{R}$, $n\mathrm{\ge }\mathrm{1}$, are independent identically distributed $L^{\mathrm{1}}$ random variables, then for almost all $\omega \mathrm{\in }{\mathrm{\Omega }}^{\mathrm{\prime }}$,

$$\frac{1}{n}\sum _{k=1}^n{X}_k(\omega )\to E(X_1).$$

Proof.

For $n\ge 0$, let $Y_n=X_{n+1}$; we do this to make the index set the same as for ${\mathrm{\Omega }}_n$. Let ${\mu }_n=Y_n{}_{*}{P}^{\prime }$ and set $\mu ={\mu }_0$. Because the $Y_n$ are identically distributed, ${\mu }_n={\mu }_0$ for each $n$.

Because $Y_0$ is $L^1$, $\mu$ has finite first moment. As ${\mu }_0=Y_0{}_{*}{P}^{\prime }$, applying the change of variables theorem (2),

$${\int }_{ℝ}t𝑑\mu (t)={\int }_{ℝ}td(Y_0{}_{*}{P}^{\prime })(t)={\int }_{{\mathrm{\Omega }}^{\prime }}{Y}_0(\omega )𝑑P^{\prime }(\omega )=E(Y_0).$$

With this, Lemma 10 says that for almost all $\omega \in \mathrm{\Omega }$,

$$\frac{1}{n}\sum _{k=0}^{n-1}{\pi }_k(\omega )\to E(Y_0).$$ (3)

For each $n$,

$${({\pi }_0,\mathrm{\dots },{\pi }_n)}_{*}P={\mu }_0\times \mathrm{\cdots }\times {\mu }_n,$$

and because the $Y_n$ are independent,

$${(Y_0,\mathrm{\dots },Y_n)}_{*}{P}^{\prime }=Y_0{}_{*}{P}^{\prime }\times \mathrm{\cdots }\times Y_n{}_{*}{P}^{\prime }={\mu }_0\times \mathrm{\cdots }\times {\mu }_n,$$

so the sequences ${\pi }_n$ and $Y_n$ have the same finite-dimensional distributions. For $n\ge 1$, define ${\mathrm{\Phi }}_n:{ℝ}^n\to ℝ$ by

$${\mathrm{\Phi }}_n(t_1,\mathrm{\dots },t_n)=\frac{1}{n}\sum _{k=1}^n{t}_k.$$

Lemma 11 then tells us that the sequences

$$U_n={\mathrm{\Phi }}_n(Y_0,\mathrm{\dots },Y_{n-1})=\frac{1}{n}\sum _{k=0}^{n-1}{Y}_k$$

and

$$V_n={\mathrm{\Phi }}_n({\pi }_0,\mathrm{\dots },{\pi }_{n-1})=\frac{1}{n}\sum _{k=0}^{n-1}{\pi }_k,$$

$n\ge 1$, have the same finite-dimensional distributions.

Now, (3) says that $V_n$ converges to $E(Y_0)$ almost surely, and thus applying Lemma 12 we get that $U_n$ converges to $E(Y_0)$ almost surely. That is,

$$\frac{1}{n}\sum _{k=0}^{n-1}{Y}_k\to E(Y_0)$$

almost surely, and because $Y_k=X_{k+1}$,

$$\frac{1}{n}\sum _{k=1}^n{X}_k\to E(X_1)$$

almost surely, completing the proof. ∎