Research IX) CLT – Central limit theorem

Research IX) CLT – Central limit theorem

In probability theory, the central limit theorem (CLT) establishes that, when independent random variables are added, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.

This theorem is a key concept in probability theory, because it implies that probabilistic and statistical methods that work for normal distributions can be applicable to many problems involving other types of distributions.

The central limit theorem has a number of variants. In its common form, the random variables must be identically distributed. In variants, convergence of the mean to the normal distribution also occurs for non-identical distributions or for non-independent observations, given that they comply with certain conditions.

The earliest version of this theorem, that the normal distribution may be used as an approximation to the binomial distribution, is now known as the de Moivre-Laplace theorem.

We’ll now try to list different approaches to the CLT.

  • Classical CLT

    Let {X1, …, Xn} be a random sample of size n — that is, a sequence of independent and identically distributed random variables drawn from distributions of expected values given by µ and finite variances given by σ2. Suppose we are interested in the sample average of these random variables.

    S_{n}:={\frac {X_{1}+\cdots +X_{n}}{n}}

    By the law of large numbers, the sample averages converge in probability to the expected value µ as n → ∞.

    The classical central limit theorem describes the size and the distributional form of the stochastic fluctuations around the deterministic number µ during this convergence.

More precisely, it states that as n gets larger, the distribution of the difference between the sample average Sn and its limit µ approximates the normal distribution with mean 0 and variance σ2. For large enough n, the distribution of Sn is close to the normal distribution with mean µ and variance σ2/n.

  • CLT under weak dependence

    A useful generalization of a sequence of independent, identically distributed random variables is a mixing random process in discrete time; this means that random variables that are temporally far apart from one another, are nearly independent. Several kinds of mixing are used in probability theory, especially strong mixing (also called α-mixing) defined by α(n) → 0 where α(n) is so-called strong mixing coefficient.

    A simplified formulation of the central limit theorem under strong mixing is:

    Theorem. Suppose that X1, X2, … is stationary and α-mixing with αn = O(n−5) and that E(Xn) = 0 and E((X^12)n) < ∞.
    Denote Sn = X1 + … + Xn, then the limit

    {\displaystyle \sigma ^{2}=\lim _{n}{\frac {\operatorname {E} \left(S_{n}^{2}\right)}{n}}}

    exists, and if σ ≠ 0 then Sn/σ√n converges in distribution to N(0,1).

    In fact, we have{\displaystyle \sigma ^{2}=\operatorname {E} \left(X_{1}^{2}\right)+2\sum _{k=1}^{\infty }\operatorname {E} \left(X_{1}X_{1+k}\right),}

    where the series converges absolutely.
    The assumption σ ≠ 0 cannot be omitted, since the asymptotic normality fails for Xn = Yn − Yn − 1 where Yn are another stationary sequence.

    There is also a stronger version of the theorem: the assumption E((X^12)
    n) < ∞     is replaced with E(|Xn|2 + δ) < ∞, and the assumption αn = O(n−5) is replaced with \sum _{n}\alpha _{n}^{\frac {\delta }{2(2+\delta )}}<\infty .

    Existence of such δ > 0 ensures the conclusion.

For a theorem of such fundamental importance to statistics and probability, the CTL has a remarkably simple proof using characteristic functions. It is similar to the proof of the (weak) law of large numbers.
We already talked about the law of large numbers (LLN). Soon, we’ll talk about the weak version of it too.

Meanwhile, as stated above, suppose {X1, …, Xn} are independent and identically distributed random variables, each with mean µ and finite variance σ2. The sum X1 + … + Xn has mean nµ and variance nσ2. Consider the random variable

{\displaystyle Z_{n}\ =\ {\frac {X_{1}+\cdots +X_{n}-n\mu }{\sqrt {n\sigma ^{2}}}}\ =\ \sum _{i=1}^{n}{\frac {X_{i}-\mu }{\sqrt {n\sigma ^{2}}}}\ =\ \sum _{i=1}^{n}{\frac {1}{\sqrt {n}}}Y_{i},}

where in the last step we defined the new random variables Yi = Xi − μ/σ, each with zero mean and unit variance (var(Y) = 1). The characteristic function of Zn is given by

{\displaystyle \varphi _{Z_{n}}\!(t)\ =\ \varphi _{\sum _{i=1}^{n}{{\frac {1}{\sqrt {n}}}Y_{i}}}\!(t)\ =\ \varphi _{Y_{1}}\!\!\left({\frac {t}{\sqrt {n}}}\right)\varphi _{Y_{2}}\!\!\left({\frac {t}{\sqrt {n}}}\right)\cdots \varphi _{Y_{n}}\!\!\left({\frac {t}{\sqrt {n}}}\right)\ =\ \left[\varphi _{Y_{1}}\!\!\left({\frac {t}{\sqrt {n}}}\right)\right]^{n},}

where in the last step we used the fact that all of the Yi are identically distributed. The characteristic function of Y1 is, by Taylor’s theorem,

{\displaystyle \varphi _{Y_{1}}\!\!\left({\frac {t}{\sqrt {n}}}\right)\ =\ 1-{\frac {t^{2}}{2n}}+c{\frac {t^{3}}{6n^{\frac {3}{2}}}}+o\!\!\left({\frac {t^{3}}{n^{\frac {3}{2}}}}\right),\quad {\bigg (}{\frac {t}{\sqrt {n}}}{\bigg )}\rightarrow 0}

where c is a (complex) constant and o(t3) is “little o notation” for some function of t that goes to zero more rapidly than t3. By the limit of the exponential function (ex= lim(1 + x/n)n), the characteristic function of Zn equals

{\displaystyle \varphi _{Z_{n}}(t)=\left(1-{\frac {t^{2}}{2n}}+c{\frac {t^{3}}{6n^{\frac {3}{2}}}}+o\left({\frac {t^{3}}{n^{\frac {3}{2}}}}\right)\right)^{n}\rightarrow e^{-{\frac {1}{2}}t^{2}},\quad n\rightarrow \infty .}

Note that all of the higher order terms vanish in the limit n → ∞. The right hand side equals the characteristic function of a standard normal distribution N(0,1), which implies that the distribution of Zn will approach N(0,1) as n → ∞.

Therefore, the sum X1 + … + Xn will approach that of the normal distribution N(nµ, nσ2), and the sample average converges to the normal distribution N(µ,σ2/n), from which the central theorem follows.{\displaystyle S_{n}={\frac {X_{1}+\cdots +X_{n}}{n}}}

The law of large numbers as well as the central limit theorem are partial solutions to a general problem: “What is the limiting behaviour of Sn as n approaches infinity?”

Suppose we have an asymptotic expansion of f(n):

{\displaystyle f(n)=a_{1}\varphi _{1}(n)+a_{2}\varphi _{2}(n)+O{\big (}\varphi _{3}(n){\big )}\qquad (n\rightarrow \infty ).}

Dividing both parts by φ1(n) and taking the limit will produce a1, the coefficient of the highest-order term in the expansion, which represents the rate at which f(n)changes in its leading term.

\lim _{n\to \infty }{\frac {f(n)}{\varphi _{1}(n)}}=a_{1}.

So f(n) grows approximately as a1φ1(n). Taking the difference between f(n) and its approximation and then dividing by the next term in the expansion, we arrive at a more refined statement about f(n):

\lim _{n\to \infty }{\frac {f(n)-a_{1}\varphi _{1}(n)}{\varphi _{2}(n)}}=a_{2}.

Here one can say that the difference between the function and its approximation grows approximately as a2φ2(n). The idea is that dividing the function by appropriate normalizing functions, and looking at the limiting behavior of the result, can tell us much about the limiting behavior of the original function itself.

Informally, when the sum Sn of independent identically distributed random variables X1, …, Xn is studied in classical probability theory, if each Xi has finite mean μ, then by the law of large numbers, Sn/n → μ, while if in addition each Xi has finite variance σ2, then by the central limit theorem, we have that:

{\frac {S_{n}-n\mu }{\sqrt {n}}}\rightarrow \xi ,

where ξ is distributed as N(0,σ2). This provides values of the first two constants in the informal expansion {\displaystyle S_{n}\approx \mu n+\xi {\sqrt {n}}.}

In the case where the Xi do not have finite mean or variance, convergence of the shifted and rescaled sum can also occur with different centering and scaling factors:{\frac {S_{n}-a_{n}}{b_{n}}}\rightarrow \Xi , or informally: {\displaystyle S_{n}\approx a_{n}+\Xi b_{n}.}

 

Lascia un commento