Law of Large Numbers

Let \(X_1, X_2, \ldots\) be independent, identically distributed random variables with \(\mu = \mathbb{E}[X_1] = \mathbb{E}[X_2] = \ldots\). Then, \[\frac{1}{n}\sum_{i=1}^{n} X_i \overset{p}{\rightarrow} \mu\] where \(\overset{p}{\rightarrow}\) indicates convergence in probability. That is, for every \(\epsilon > 0\), \[\lim_{n\rightarrow\infty} P\left(\left\lvert \frac{1}{n}\sum_{i=1}^{n}X_i - \mu \right\rvert > \epsilon \right) = 0\]

Let \(\bar{X} = \sum_{i=1}^{n}X_i\). The strong law says that \(\bar{X}\) converges to \(\mu\) almost surely. That is, \[ P\left(\lim_{n\rightarrow\infty} \bar{X} \rightarrow \mu \right) = 1 \]

The difference between the strong and weak law comes from the difference in the type of convergence that they use. The weak law says that for any \(\epsilon\) margin around \(\mu\), no matter how small, we can be arbitrarily confident that \(\bar{X}\) lies within the margin after a certain \(n_0\). Note that this does not forbid \(\bar{X}\) from leaving this margin infinitely many times after \(n_0\).

In contrast, the strong law says that with probability 1, there exists an \(n_0\), beyond which \(\bar{X}\) never leaves \(\epsilon\).

A consequence of the law of large numbers is the fact that samples \(X_i\) plotted in a scaled histogram will visually approach the density function of \(X\).

Make bins, each of width \(h\). Let \(a_1, a_{2},\ldots\) be the boundary points for these bins. If we take \(n\) samples, \(X_1,\ldots X_n\), then the number of fraction that fall in the \(k\) th bin is: \[n_k = \frac{1}{n} \sum_{i=1}^{n}I_{a_k < X_i \leq a_{k+1}}\] But by the law of large numbers, we know that \(n_k \rightarrow P(a_k < X_i \leq a_{k+1})\). That is, the height of the \(k\) th bin will trend to the probability of a sample landing in that bin as \(n\) goes to infinity.

• UMD histograms and LLN