|
|
| Line 1: |
Line 1: |
| In [[probability theory]], there exist several different notions of '''convergence of random variables'''. The [[limit of a sequence|convergence]] of [[sequence]]s of [[random variable]]s to some [[Limit of a sequence|limit]] random variable is an important concept in probability theory, and its applications to [[statistics]] and [[stochastic process]]es. The same concepts are known in more general [[mathematics]] as '''stochastic convergence''' and they formalize the idea that a sequence of essentially random or unpredictable events can sometimes be expected to settle down into a behaviour that is essentially unchanging when items far enough into the sequence are studied. The different possible notions of convergence relate to how such a behaviour can be characterised: two readily understood behaviours are that the sequence eventually takes a constant value, and that values in the sequence continue to change but can be described by an unchanging probability distribution.
| | Nice to meet you, i'm Neda Kester. Distributing production is what she does and it is something she actually enjoy. Years ago he moved to Kansas. My friends say it's not good for me but what Appreciate doing is [https://www.google.com/search?hl=en&gl=us&tbm=nws&q=mountain+biking mountain biking] but I'm thinking on starting something more [http://Www.Dailymail.Co.uk/home/search.html?sel=site&searchPhrase=challenging challenging]. His wife and he have a website. You may decide to check it out: http://www.eemafrance.fr/<br><br>Feel free to visit my homepage - zanotti homme solde - [http://www.eemafrance.fr/ www.eemafrance.fr], |
| | |
| ==Background==
| |
| | |
| "Stochastic convergence" formalizes the idea that a sequence of essentially random or unpredictable events can sometimes be expected to settle into a pattern. The pattern may for instance be
| |
| *[[Limit of a sequence|Convergence]] in the classical sense to a fixed value, perhaps itself coming from a random event
| |
| *An increasing similarity of outcomes to what a purely deterministic function would produce
| |
| *An increasing preference towards a certain outcome
| |
| *An increasing "aversion" against straying far away from a certain outcome
| |
| | |
| Some less obvious, more theoretical patterns could be
| |
| *That the probability distribution describing the next outcome may grow increasingly similar to a certain distribution
| |
| *That the series formed by calculating the [[expected value]] of the outcome's distance from a particular value may converge to 0
| |
| *That the variance of the [[random variable]] describing the next event grows smaller and smaller.
| |
| These other types of patterns that may arise are reflected in the different types of stochastic convergence that have been studied.
| |
| | |
| While the above discussion has related to the convergence of a single series to a limiting value, the notion of the convergence of two series towards each other is also important, but this is easily handled by studying the sequence defined as either the difference or the ratio of the two series.
| |
| | |
| For example, if the average of ''n'' [[Independence_(probability_theory)|independent]] random variables ''Y''<sub>''i''</sub>, ''i'' = 1, ..., ''n'', all having the same finite [[mean]] and [[variance]], is given by
| |
| | |
| :<math>X_n = \frac{1}{n}\sum_{i=1}^n Y_i\,,</math>
| |
| | |
| then as ''n'' tends to infinity, ''X''<sub>''n''</sub> converges ''in probability'' (see below) to the common [[mean]], μ, of the random variables ''Y''<sub>''i''</sub>. This result is known as the [[weak law of large numbers]]. Other forms of convergence are important in other useful theorems, including the [[central limit theorem]].
| |
| | |
| Throughout the following, we assume that (''X''<sub>''n''</sub>) is a sequence of random variables, and ''X'' is a random variable, and all of them are defined on the same [[probability space]] <math>\scriptstyle (\Omega, \mathcal{F}, P )</math>.
| |
| | |
| == Convergence in distribution ==
| |
| {{Infobox
| |
| | title = Examples of convergence in distribution
| |
| | bodystyle = width: 28em;
| |
| | headerstyle = background-color: lightblue; text-align: left; padding-left: 3pt;
| |
| | datastyle = text-align: left;
| |
| | header1 = Dice factory
| |
| | data2 = Suppose a new dice factory has just been built. The first few dice come out quite biased, due to imperfections in the production process. The outcome from tossing any of them will follow a distribution markedly different from the desired [[uniform distribution (discrete)|uniform distribution]]. <br /><br />As the factory is improved, the dice become less and less loaded, and the outcomes from tossing a newly produced dice will follow the uniform distribution more and more closely.
| |
| | header3 = Tossing coins
| |
| | data4 = Let ''X<sub>n</sub>'' be the fraction of heads after tossing up an unbiased coin ''n'' times. Then ''X''<sub>1</sub> has the [[Bernoulli distribution]] with expected value ''μ'' = 0.5 and variance ''σ''<sup>2</sup> = 0.25. The subsequent random variables ''X''<sub>2</sub>, ''X''<sub>3</sub>, … will all be distributed [[Binomial distribution|binomially]].<br /><br />As ''n'' grows larger, this distribution will gradually start to take shape more and more similar to the [[Normal distribution|bell curve]] of the normal distribution. If we shift and rescale ''X<sub>n</sub>''’s appropriately, then <math>\scriptstyle Z_n = \sqrt{n}(X_n-\mu)/\sigma</math> will be '''converging in distribution''' to the standard normal, the result that follows from the celebrated [[central limit theorem]].
| |
| | header5 = Graphic example
| |
| | data6 = Suppose { ''X''<sub>''i''</sub> } is an [[Independent and identically distributed|iid]] sequence of [[uniform distribution (continuous)|uniform]] ''U''(−1,1) random variables. Let <math>\scriptstyle Z_n = {\scriptscriptstyle\frac{1}{\sqrt{n}}}\sum_{i=1}^n X_i</math> be their (normalized) sums. Then according to the [[central limit theorem]], the distribution of ''Z<sub>n</sub>'' approaches the normal ''N''(0, ⅓) distribution. This convergence is shown in the picture: as ''n'' grows larger, the shape of the pdf function gets closer and closer to the Gaussian curve.
| |
| [[Image:Convergence in distribution (sum of uniform rvs).gif|center|200px]]
| |
| }}
| |
| | |
| With this mode of convergence, we increasingly expect to see the next outcome in a sequence of random experiments becoming better and better modeled by a given [[probability distribution]].
| |
| | |
| Convergence in distribution is the weakest form of convergence, since it is implied by all other types of convergence mentioned in this article. However convergence in distribution is very frequently used in practice; most often it arises from application of the [[central limit theorem]].
| |
| | |
| ===Definition===
| |
| A sequence <math>X_1,\ X_2,\ldots</math> of [[random variable]]s is said to '''converge in distribution''', or '''converge weakly''', or '''converge in law''' to a random variable ''X'' if
| |
| | |
| : <math>
| |
| \lim_{n\to\infty} F_n(x) = F(x),
| |
| </math>
| |
| | |
| for every number {{nowrap|''x'' ∈ '''R'''}} at which <math>F</math> is [[continuous function|continuous]]. Here <math>F_n</math> and <math>F</math> are the [[cumulative distribution function]]s of random variables <math>X_n</math> and <math>X</math>, respectively.
| |
| | |
| The requirement that only the continuity points of <math>F</math> should be considered is essential. For example if <math>X_n</math> are distributed [[Uniform distribution (continuous)|uniformly]] on intervals [0, {{frac|''n''}}], then this sequence converges in distribution to a [[degenerate distribution|degenerate]] random variable {{nowrap|''X'' {{=}} 0}}. Indeed, {{nowrap|''F<sub>n</sub>''(''x'') {{=}} 0}} [[existential quantification|for all]] ''n'' when {{nowrap|''x'' ≤ 0}}, and {{nowrap|''F<sub>n</sub>''(''x'') {{=}} 1}} for all {{nowrap|''x'' ≥ {{frac|''n''}}}} when {{nowrap|''n'' > 0}}. However, for this limiting random variable {{nowrap|''F''(0) {{=}} 1}}, even though {{nowrap|''F<sub>n</sub>''(0) {{=}} 0}} for all ''n''. Thus the convergence of cdfs fails at the point {{nowrap|''x'' {{=}} 0}} where ''F'' is discontinuous.
| |
| | |
| Convergence in distribution may be denoted as
| |
| : <math>\begin{align}
| |
| & X_n \ \xrightarrow{d}\ X,\ \
| |
| X_n \ \xrightarrow{\mathcal{D}}\ X,\ \
| |
| X_n \ \xrightarrow{\mathcal{L}}\ X,\ \
| |
| X_n \ \xrightarrow{d}\ \mathcal{L}_X, \\
| |
| & X_n \rightsquigarrow X,\ \
| |
| X_n \Rightarrow X,\ \
| |
| \mathcal{L}(X_n)\to\mathcal{L}(X),\\
| |
| \end{align}</math>
| |
| where <math>\scriptstyle\mathcal{L}_X</math> is the law (probability distribution) of <math>X</math>. For example if <math>X</math> is standard normal we can write <math style="height:1.5em;position:relative;top:-.3em">X_n\,\xrightarrow{d}\,\mathcal{N}(0,\,1)</math>.
| |
| | |
| For [[random vector]]s {''X''<sub>1</sub>, ''X''<sub>2</sub>, …} ⊂ '''R'''<sup>''k''</sup> the convergence in distribution is defined similarly. We say that this sequence '''converges in distribution''' to a random ''k''-vector ''X'' if
| |
| : <math>
| |
| \lim_{n\to\infty} \operatorname{Pr}(X_n\in A) = \operatorname{Pr}(X\in A)
| |
| </math>
| |
| for every ''A'' ⊂ '''R'''<sup>''k''</sup> which is a [[continuity set]] of ''X''.
| |
| | |
| The definition of convergence in distribution may be extended from random vectors to more general [[random element]]s in arbitrary [[metric space]]s, and even to the “random variables” which are not measurable — a situation which occurs for example in the study of [[empirical process]]es. This is the “weak convergence of laws without laws being defined” — except asymptotically.<ref>{{harvnb|Bickel|Klaassen|Ritov|Wellner|1998|loc=A.8, page 475}}</ref>
| |
| | |
| In this case the term '''weak convergence''' is preferable (see [[weak convergence of measures]]), and we say that a sequence of random elements {''X<sub>n</sub>''} converges weakly to ''X'' (denoted as ''X<sub>n</sub>'' ⇒ ''X'') if
| |
| : <math> | |
| \operatorname{E}^*h(X_n) \to \operatorname{E}\,h(X)
| |
| </math>
| |
| for all continuous bounded functions ''h''(·).<ref>{{harvnb|van der Vaart|Wellner|1996|page=4}}</ref> Here E* denotes the ''outer expectation'', that is the expectation of a “smallest measurable function ''g'' that dominates ''h''(''X''<sub>''n''</sub>)”.
| |
| | |
| ===Properties===
| |
| | |
| * Since ''F''(''a'') = Pr(''X'' ≤ ''a''), the convergence in distribution means that the probability for ''X<sub>n</sub>'' to be in a given range is approximately equal to the probability that the value of ''X'' is in that range, provided ''n'' is [[sufficiently large]].
| |
| | |
| * In general, convergence in distribution does not imply that the sequence of corresponding [[probability density function]]s will also converge. As an example one may consider random variables with densities ''ƒ<sub>n</sub>''(''x'') = (1 − cos(2''πnx''))'''1'''<sub>{''x''∈(0,1)}</sub>. These random variables converge in distribution to a uniform ''U''(0, 1), whereas their densities do not converge at all.<ref>{{harvnb|Romano|Siegel|1985|loc=Example 5.26}}</ref>
| |
| ** However, [[Scheffé’s lemma]] implies that convergence of the [[probability density function]]s implies convergence in distribution.<ref name=koro>{{cite web|last=Koro|title=Scheffé's theorem|url=http://planetmath.org/ScheffesTheorem.html|accessdate=1 February 2013}}</ref>
| |
| | |
| * The '''[[portmanteau lemma]]''' provides several equivalent definitions of convergence in distribution. Although these definitions are less intuitive, they are used to prove a number of statistical theorems. The lemma states that {''X''<sub>''n''</sub>} converges in distribution to ''X'' if and only if any of the following statements are true:{{citation needed|date=February 2013}}
| |
| * Eƒ(''X<sub>n</sub>'') → Eƒ(''X'') for all [[Bounded function|bounded]], [[continuous function]]s ƒ (where E denotes the [[expected value]]);
| |
| * Eƒ(''X<sub>n</sub>'') → Eƒ(''X'') for all bounded, [[Lipschitz function]]s ƒ;
| |
| * limsup{ Eƒ(''X<sub>n</sub>'') } ≤ Eƒ(''X'') for every [[upper semi-continuous]] function ƒ bounded from above;
| |
| * liminf{ Eƒ(''X<sub>n</sub>'') } ≥ Eƒ(''X'') for every [[lower semi-continuous]] function ƒ bounded from below;
| |
| * limsup{ Pr(''X<sub>n</sub>'' ∈ ''C'') } ≤ Pr(''X'' ∈ ''C'') for all [[closed set]]s ''C'';
| |
| * liminf{ Pr(''X<sub>n</sub>'' ∈ ''U'') } ≥ Pr(''X'' ∈ ''U'') for all [[open set]]s ''U'';
| |
| * lim{ Pr(''X<sub>n</sub>'' ∈ ''A'') } = Pr(''X'' ∈ ''A'') for all [[continuity set]]s ''A'' of random variable ''X''.
| |
| | |
| * The '''[[continuous mapping theorem]]''' states that for a continuous function ''g''(·), if the sequence {''X''<sub>''n''</sub>} converges in distribution to ''X'', then {''g''(''X''<sub>''n''</sub>)} converges in distribution to ''g''(''X'').
| |
| ** Note however that convergence in distribution of {''X''<sub>''n''</sub>} to ''X'' and {''Y''<sub>''n''</sub>} to ''Y'' does in general ''not'' imply convergence in distribution of {''X''<sub>''n''</sub>+''Y''<sub>''n''</sub>} to ''X''+''Y'' or of {''X''<sub>''n''</sub>''Y''<sub>''n''</sub>} to ''XY''.
| |
| | |
| * '''[[Lévy’s continuity theorem]]:''' the sequence {''X''<sub>''n''</sub>} converges in distribution to ''X'' if and only if the sequence of corresponding [[characteristic function (probability theory)|characteristic function]]s {''φ<sub>n</sub>''} [[pointwise convergence|converges pointwise]] to the characteristic function ''φ'' of ''X''.
| |
| | |
| * Convergence in distribution is [[metrizable]] by the [[Lévy–Prokhorov metric]].
| |
| | |
| * A natural link to convergence in distribution is the [[Skorokhod's representation theorem]].
| |
| | |
| == Convergence in probability ==
| |
| {{Infobox
| |
| | title = Examples of convergence in probability
| |
| | bodystyle = width: 28em;
| |
| | headerstyle = background-color: lightblue; text-align: left; padding-left: 3pt;
| |
| | datastyle = text-align: left;
| |
| | header1 = Height of a person
| |
| | data2 = Consider the following experiment. First, pick a random person in the street. Let ''X'' be his/her height, which is ''ex ante'' a random variable. Then you start asking other people to estimate this height by eye. Let ''X<sub>n</sub>'' be the average of the first ''n'' responses. Then (provided there is no [[systematic error]]) by the [[law of large numbers]], the sequence ''X<sub>n</sub>'' will converge in probability to the random variable ''X''.
| |
| | header3 = Archer
| |
| | data4 = Suppose a person takes a bow and starts shooting arrows at a target. Let ''X<sub>n</sub>'' be his score in ''n''-th shot. Initially he will be very likely to score zeros, but as the time goes and his [[archery]] skill increases, he will become more and more likely to hit the [[bullseye (target)|bullseye]] and score 10 points. After the years of practice the probability that he hit anything but 10 will be getting increasingly smaller and smaller and will converge to 0. Thus, the sequence ''X<sub>n</sub>'' converges in probability to ''X'' = 10.<br /><br />Note that ''X<sub>n</sub>'' does not converge almost surely however. No matter how professional the archer becomes, there will always be a small probability of making an error. Thus the sequence {''X<sub>n</sub>''} will never turn stationary: there will always be non-perfect scores in it, even if they are becoming increasingly less frequent.
| |
| }}
| |
| | |
| The basic idea behind this type of convergence is that the probability of an “unusual” outcome becomes smaller and smaller as the sequence progresses.
| |
| | |
| The concept of convergence in probability is used very often in statistics. For example, an estimator is called [[consistent estimator|consistent]] if it converges in probability to the quantity being estimated. Convergence in probability is also the type of convergence established by the [[weak law of large numbers]].
| |
| | |
| === Definition ===
| |
| A sequence {''X''<sub>''n''</sub>} of random variables '''converges in probability''' towards the random variable ''X'' if for all ''ε'' > 0
| |
| : <math>
| |
| \lim_{n\to\infty}\Pr\big(|X_n-X| \geq \varepsilon\big) = 0.
| |
| </math>
| |
| | |
| Formally, pick any ''ε'' > 0 and any ''δ'' > 0. Let ''P''<sub>''n''</sub> be the probability that ''X<sub>n</sub>'' is outside the ball of radius ''ε'' centered at ''X''. Then for ''X''<sub>''n''</sub> to converge in probability to ''X'' there should exist a number ''N'' (which will depend on ''ε'' and ''δ'') such that for all ''n'' ≥ ''N'' the probability ''P''<sub>''n''</sub> is less than ''δ''.
| |
| | |
| Convergence in probability is denoted by adding the letter ''p'' over an arrow indicating convergence, or using the “plim” probability limit operator:
| |
| : <math>
| |
| X_n \ \xrightarrow{p}\ X,\ \
| |
| X_n \ \xrightarrow{P}\ X,\ \
| |
| \underset{n\to\infty}{\operatorname{plim}}\, X_n = X.
| |
| </math>
| |
| | |
| For random elements {''X''<sub>''n''</sub>} on a [[separable metric space]] (''S'', ''d''), convergence in probability is defined similarly by<ref>{{harvnb|Dudley|2002|loc=Chapter 9.2, page 287}}</ref>
| |
| : <math>
| |
| \forall\varepsilon>0, \Pr\big(d(X_n,X)\geq\varepsilon\big) \to 0.
| |
| </math>
| |
| | |
| ===Properties===
| |
| | |
| * Convergence in probability implies convergence in distribution.<sup>[[Proofs of convergence of random variables#propA2|[proof]]]</sup>
| |
| | |
| * Convergence in probability does not imply almost sure convergence.<sup>[[Proofs of convergence of random variables#propA1i|[proof]]]</sup>
| |
| | |
| * In the opposite direction, convergence in distribution implies convergence in probability only when the limiting random variable ''X'' is a constant.<sup>[[Proofs of convergence of random variables#propB1|[proof]]]</sup>
| |
| | |
| * The [[continuous mapping theorem]] states that for every continuous function ''g''(·), if  <math style="position:relative;top:-.4em">\scriptstyle X_n\xrightarrow{p}X</math>, then also  <math style="position:relative;top:-.35em">\scriptstyle g(X_n)\xrightarrow{p}g(X)</math>.
| |
| | |
| * Convergence in probability defines a [[topology]] on the space of random variables over a fixed probability space. This topology is [[metrizable]] by the ''[[Ky Fan]] metric'':<ref>{{harvnb|Dudley|2002|page=289}}</ref>
| |
| : <math style="position:relative;top:.3em">
| |
| d(X,Y) = \inf\!\big\{ \varepsilon>0:\ \Pr\big(|X-Y|>\varepsilon\big)\leq\varepsilon\big\}
| |
| </math>
| |
| or
| |
| : <math>d(X,Y)=\mathbb E\left[\min(|X-Y|, 1)\right]</math>.
| |
| | |
| == Almost sure convergence ==
| |
| {{Infobox
| |
| | title = Examples of almost sure convergence
| |
| | bodystyle = width: 28em;
| |
| | headerstyle = background-color: lightblue; text-align:left;
| |
| | datastyle = text-align: left;
| |
| | header1 = Example 1
| |
| | data2 = Consider an animal of some short-lived species. We record the amount of food that this animal consumes per day. This sequence of numbers will be unpredictable, but we may be ''quite certain'' that one day the number will become zero, and will stay zero forever after.
| |
| | header3 = Example 2
| |
| | data4 = Consider a man who tosses seven coins every morning. Each afternoon, he donates one pound to a charity for each head that appeared. The first time the result is all tails, however, he will stop permanently.<br /><br />Let ''X''<sub>1</sub>, ''X''<sub>2</sub>, … be the daily amounts the charity receives from him.<br /><br />We may be ''almost sure'' that one day this amount will be zero, and stay zero forever after that.<br /><br />However, when we consider ''any finite number'' of days, there is a nonzero probability the terminating condition will not occur.
| |
| }}
| |
| | |
| This is the type of stochastic convergence that is most similar to [[pointwise convergence]] known from elementary [[real analysis]].
| |
| | |
| ===Definition===
| |
| To say that the sequence ''X''<sub>''n''</sub> converges '''almost surely''' or '''almost everywhere''' or '''with probability 1''' or '''strongly''' towards ''X'' means that
| |
| : <math> | |
| \operatorname{Pr}\!\left( \lim_{n\to\infty}\! X_n = X \right) = 1.
| |
| </math>
| |
| | |
| This means that the values of ''X''<sub>''n''</sub> approach the value of ''X'', in the sense (see [[almost surely]]) that events for which ''X''<sub>''n''</sub> does not converge to ''X'' have probability 0. Using the probability space <math>\scriptstyle (\Omega, \mathcal{F}, Pr )</math> and the concept of the random variable as a function from Ω to '''R''', this is equivalent to the statement
| |
| : <math>
| |
| \operatorname{Pr}\Big( \omega \in \Omega : \lim_{n \to \infty} X_n(\omega) = X(\omega) \Big) = 1.
| |
| </math>
| |
| | |
| Another, equivalent, way of defining almost sure convergence is as follows:
| |
| : <math>
| |
| \lim_{n\to\infty} \operatorname{Pr}\Big( \omega \in \Omega : \sup_{m\ge n} | X_m(\omega) - X(\omega) | \ge \varepsilon \Big) = 0 \quad\text{for all}\quad \varepsilon>0.
| |
| </math>
| |
| | |
| Almost sure convergence is often denoted by adding the letters ''a.s.'' over an arrow indicating convergence:
| |
| : <math>
| |
| X_n \, \xrightarrow{\mathrm{a.s.}} \, X.
| |
| </math>
| |
| | |
| For generic [[random element]]s {''X<sub>n</sub>''} on a [[metric space]] (''S'', ''d''), convergence almost surely is defined similarly:
| |
| : <math>
| |
| \operatorname{Pr}\Big( \omega\in\Omega:\, d\big(X_n(\omega),X(\omega)\big)\,\underset{n\to\infty}{\longrightarrow}\,0 \Big) = 1
| |
| </math>
| |
| | |
| ===Properties===
| |
| * Almost sure convergence implies convergence in probability, and hence implies convergence in distribution. It is the notion of convergence used in the strong [[law of large numbers]].
| |
| * The concept of almost sure convergence does not come from a [[topology]] on the space of random variables. This means there is no topology on the space of random variables such that the almost surely convergent sequences are exactly the converging sequences with respect to that topology. In particular, there is no metric of almost sure convergence.
| |
| | |
| == Sure convergence ==
| |
| | |
| To say that the sequence of [[random variables]] (''X''<sub>''n''</sub>) defined over the same [[probability space]] (i.e., a [[random process]]) converges '''surely''' or '''everywhere''' or '''pointwise''' towards ''X'' means
| |
| :<math>\lim_{n\rightarrow\infty}X_n(\omega)=X(\omega), \, \, \forall \omega \in \Omega.</math>
| |
| where Ω is the [[sample space]] of the underlying [[probability space]] over which the random variables are defined.
| |
| | |
| This is the notion of [[pointwise convergence]] of sequence functions extended to sequence of [[random variables]]. (Note that random variables themselves are functions).
| |
| | |
| :<math>\big\{\omega \in \Omega \, | \, \lim_{n \to \infty}X_n(\omega) = X(\omega) \big\} = \Omega.</math>
| |
| | |
| Sure convergence of a random variable implies all the other kinds of convergence stated above, but there is no payoff in [[probability theory]] by using sure convergence compared to using almost sure convergence. The difference between the two only exists on sets with probability zero. This is why the concept of sure convergence of random variables is very rarely used.
| |
| | |
| == Convergence in mean ==
| |
| | |
| Given a real number {{nowrap|''r'' ≥ 1}}, we say that the sequence ''X<sub>n</sub>'' converges '''in the ''r''-th mean''' (or '''in the [[Lp space|''L<sup>r</sup>''-norm]]''') towards the random variable ''X'', if the {{nowrap|''r''-th}} [[Moment (mathematics)|absolute moment]]s E(|''X<sub>n</sub>''|<sup>''r''</sup>) and E(|''X''|<sup>''r''</sup>) of ''X<sub>n</sub>'' and ''X'' exist, and
| |
| : <math> | |
| \lim_{n\to\infty} \operatorname{E}\left( |X_n-X|^r \right) = 0,
| |
| </math>
| |
| where the operator E denotes the [[expected value]]. Convergence in {{nowrap|''r''-th}} mean tells us that the expectation of the {{nowrap|''r''-th}} power of the difference between ''X<sub>n</sub>'' and ''X'' converges to zero.
| |
| | |
| This type of convergence is often denoted by adding the letter ''L<sup>r</sup>'' over an arrow indicating convergence:
| |
| | |
| : <math>X_n \, \xrightarrow{L^r} \, X.</math>
| |
| | |
| The most important cases of convergence in ''r''-th mean are:
| |
| * When ''X''<sub>''n''</sub> converges in ''r''-th mean to ''X'' for ''r'' = 1, we say that ''X''<sub>''n''</sub> converges '''in mean''' to ''X''.
| |
| * When ''X''<sub>''n''</sub> converges in ''r''-th mean to ''X'' for ''r'' = 2, we say that ''X''<sub>''n''</sub> converges '''in mean square''' to ''X''.
| |
| | |
| Convergence in the ''r''-th mean, for ''r'' ≥ 1, implies convergence in probability (by [[Markov's inequality]]). Furthermore, if ''r'' > ''s'' ≥ 1, convergence in ''r''-th mean implies convergence in ''s''-th mean. Hence, convergence in mean square implies convergence in mean.
| |
| | |
| ==Properties==
| |
| * If <math>X_n\ \xrightarrow{p}\ X</math> and <math>X_n\ \xrightarrow{p}\ Y</math>, then <math>X=Y</math> [[almost surely]].
| |
| * If <math>X_n\ \xrightarrow{a.s.}\ X</math> and <math>X_n\ \xrightarrow{a.s.}\ Y</math>, then <math>X=Y</math> almost surely.
| |
| * If <math>X_n\ \xrightarrow{L^r}\ X</math> and <math>X_n\ \xrightarrow{L^r}\ Y</math>, then <math>X=Y</math> almost surely.
| |
| * If <math>X_n\ \xrightarrow{p}\ X</math> and <math>Y_n\ \xrightarrow{p}\ Y</math>, then <math>aX_n+bY_n\ \xrightarrow{p}\ aX+bY</math> (for any real numbers <math>a</math> and <math>b</math>) and <math>X_n Y_n\xrightarrow{p}\ XY</math>.
| |
| * If <math>X_n\ \xrightarrow{a.s.}\ X</math> and <math>Y_n\ \xrightarrow{a.s.}\ Y</math>, then <math>aX_n+bY_n\ \xrightarrow{a.s.}\ aX+bY</math> (for any real numbers <math>a</math> and <math>b</math>) and <math>X_n Y_n\xrightarrow{a.s.}\ XY</math>.
| |
| * If <math>X_n\ \xrightarrow{L^r}\ X</math> and <math>Y_n\ \xrightarrow{L^r}\ Y</math>, then <math>aX_n+bY_n\ \xrightarrow{L^r}\ aX+bY</math> (for any real numbers <math>a</math> and <math>b</math>).
| |
| * None of the above statements are true for convergence in distribution.
| |
|
| |
| The chain of implications between the various notions of convergence are noted in their respective sections. They are, using the arrow notation:
| |
| : <math>\begin{matrix}
| |
| \xrightarrow{L^s} & \underset{s>r\geq1}{\Rightarrow} & \xrightarrow{L^r} & & \\
| |
| & & \Downarrow & & \\
| |
| \xrightarrow{a.s.} & \Rightarrow & \xrightarrow{\ p\ } & \Rightarrow & \xrightarrow{\ d\ }
| |
| \end{matrix}</math>
| |
| | |
| These properties, together with a number of other special cases, are summarized in the following list:
| |
| | |
| * {{anchor|propA1}} Almost sure convergence implies convergence in probability:<ref name="vdv2">{{harvnb|van der Vaart|1998|loc=Theorem 2.7}}</ref><sup>[[Proofs of convergence of random variables#propA1|[proof]]]</sup>
| |
| : <math>
| |
| X_n\ \xrightarrow{as}\ X \quad\Rightarrow\quad X_n\ \xrightarrow{p}\ X
| |
| </math>
| |
| | |
| * {{anchor|propA1}} Convergence in probability implies there exists a sub-sequence <math>(k_n)</math> which almost surely converges:<ref>{{cite book|last=Gut|first=Allan|title=Probability: A graduate course|year=2005|publisher=Springer|location=Theorem 3.4|isbn=0-387-22833-0}}</ref>
| |
| : <math>
| |
| X_n\ \xrightarrow{p}\ X \quad\Rightarrow\quad X_{k_n}\ \xrightarrow{as}\ X
| |
| </math>
| |
| | |
| * {{anchor|propA2}} Convergence in probability implies convergence in distribution:<ref name="vdv2"/><sup>[[Proofs of convergence of random variables#propA2|[proof]]]</sup>
| |
| : <math>
| |
| X_n\ \xrightarrow{p}\ X \quad\Rightarrow\quad X_n\ \xrightarrow{d}\ X
| |
| </math>
| |
| | |
| * {{anchor|propA3}} Convergence in ''r''-th order mean implies convergence in probability:
| |
| : <math>
| |
| X_n\ \xrightarrow{L^r}\ X \quad\Rightarrow\quad X_n\ \xrightarrow{p}\ X
| |
| </math>
| |
| | |
| * {{anchor|propA4}} Convergence in ''r''-th order mean implies convergence in lower order mean, assuming that both orders are greater than or equal to one:
| |
| : <math>
| |
| X_n\ \xrightarrow{L^r}\ X \quad\Rightarrow\quad X_n\ \xrightarrow{L^s}\ X,
| |
| </math> <span style="position:relative;top:.4em;left:2em;">provided ''r'' ≥ ''s'' ≥ 1.</span>
| |
| | |
| * {{anchor|propB1}} If ''X''<sub>''n''</sub> converges in distribution to a constant ''c'', then ''X''<sub>''n''</sub> converges in probability to ''c'':<ref name="vdv2"/><sup>[[Proofs of convergence of random variables#propB1|[proof]]]</sup>
| |
| : <math>
| |
| X_n\ \xrightarrow{d}\ c \quad\Rightarrow\quad X_n\ \xrightarrow{p}\ c,
| |
| </math> <span style="position:relative;top:.4em;left:2em;">provided ''c'' is a constant.</span>
| |
| | |
| * {{anchor|propB2}} If ''X<sub>n</sub>'' converges in distribution to ''X'' and the difference between ''X<sub>n</sub>'' and ''Y<sub>n</sub>'' converges in probability to zero, then ''Y<sub>n</sub>'' also converges in distribution to ''X'':<ref name="vdv2"/><sup>[[Proofs of convergence of random variables#propB2|[proof]]]</sup>
| |
| : <math>
| |
| X_n\ \xrightarrow{d}\ X,\ \ |X_n-Y_n|\ \xrightarrow{p}\ 0\ \quad\Rightarrow\quad Y_n\ \xrightarrow{d}\ X
| |
| </math>
| |
| | |
| * {{anchor|propB3}} If ''X<sub>n</sub>'' converges in distribution to ''X'' and ''Y<sub>n</sub>'' converges in distribution to a constant ''c'', then the joint vector (''X''<sub>''n''</sub>, ''Y''<sub>''n''</sub>) converges in distribution to ''(X, c)'':<ref name="vdv2"/><sup>[[Proofs of convergence of random variables#propB3|[proof]]]</sup>
| |
| | |
| : <math>
| |
| X_n\ \xrightarrow{d}\ X,\ \ Y_n\ \xrightarrow{d}\ c\ \quad\Rightarrow\quad (X_n,Y_n)\ \xrightarrow{d}\ (X,c)
| |
| </math> <span style="position:relative;top:.4em;left:2em;">provided ''c'' is a constant.</span>
| |
| | |
| :Note that the condition that ''Y<sub>n</sub>'' converges to a constant is important, if it were to converge to a random variable ''Y'' then we wouldn’t be able to conclude that (''X''<sub>''n''</sub>, ''Y''<sub>''n''</sub>) converges to ''(X, Y)''.
| |
| | |
| * {{anchor|propB4}} If ''X<sub>n</sub>'' converges in probability to ''X'' and ''Y<sub>n</sub>'' converges in probability to ''Y'', then the joint vector (''X''<sub>''n''</sub>, ''Y''<sub>''n''</sub>) converges in probability to (''X'', ''Y''):<ref name="vdv2"/><sup>[[Proofs of convergence of random variables#propB4|[proof]]]</sup>
| |
| : <math>
| |
| X_n\ \xrightarrow{p}\ X,\ \ Y_n\ \xrightarrow{p}\ Y\ \quad\Rightarrow\quad (X_n,Y_n)\ \xrightarrow{p}\ (X,Y)
| |
| </math>
| |
| | |
| * If ''X''<sub>''n''</sub> converges in probability to ''X'', and if {{nowrap|'''P'''({{!}}''X<sub>n</sub>''{{!}} ≤ ''b'') {{=}} 1}} for all ''n'' and some ''b'', then ''X''<sub>''n''</sub> converges in ''r''th mean to ''X'' for all ''r'' ≥ 1. In other words, if ''X''<sub>''n''</sub> converges in probability to ''X'' and all random variables ''X''<sub>''n''</sub> are almost surely bounded above and below, then ''X''<sub>''n''</sub> converges to ''X'' also in any ''r''th mean.
| |
| | |
| * '''Almost sure representation'''. Usually, convergence in distribution does not imply convergence almost surely. However for a given sequence {''X<sub>n</sub>''} which converges in distribution to ''X''<sub>0</sub> it is always possible to find a new probability space (Ω, ''F'', P) and random variables {''Y<sub>n</sub>'', ''n'' = 0,1,…} defined on it such that ''Y<sub>n</sub>'' is equal in distribution to ''X<sub>n</sub>'' for each ''n'' ≥ 0, and ''Y<sub>n</sub>'' converges to ''Y''<sub>0</sub> almost surely.<ref>{{harvnb|van der Vaart|1998|loc=Th.2.19}}</ref>
| |
| | |
| * If for all ''ε'' > 0,
| |
| | |
| ::<math>\sum_n \mathbb{P} \left(|X_n - X| > \varepsilon\right) < \infty,</math>
| |
| | |
| :then we say that ''X''<sub>''n''</sub> ''converges almost completely'', or ''almost in probability'' towards ''X''. When ''X''<sub>''n''</sub> converges almost completely towards ''X'' then it also converges almost surely to ''X''. In other words, if ''X''<sub>''n''</sub> converges in probability to ''X'' sufficiently quickly (''i''.''e''. the above sequence of tail probabilities is summable for all ''ε'' > 0), then ''X''<sub>''n''</sub> also converges almost surely to ''X''. This is a direct implication from the [[Borel-Cantelli lemma]].
| |
| | |
| * If ''S''<sub>''n''</sub> is a sum of ''n'' real independent random variables:
| |
| | |
| ::<math>S_n = X_1+\cdots+X_n \, </math>
| |
| | |
| :then ''S''<sub>''n''</sub> converges almost surely if and only if ''S''<sub>''n''</sub> converges in probability.
| |
| | |
| * The [[dominated convergence theorem]] gives sufficient conditions for almost sure convergence to imply ''L''<sup>1</sup>-convergence:
| |
| ::<math>
| |
| \left. \begin{array}{ccc}
| |
| X_n\xrightarrow{a.s.} X
| |
| \\ \\
| |
| |X_n| < Y
| |
| \\ \\
| |
| \mathrm{E}(Y) < \infty
| |
| \end{array}\right\} \quad\Rightarrow \quad X_n\xrightarrow{L^1} X
| |
| </math>
| |
| | |
| *A necessary and sufficient condition for ''L''<sup>1</sup> convergence is <math>X_n\xrightarrow{P} X</math> and the sequence (''X<sub>n</sub>'') is [[uniformly integrable]].
| |
| | |
| ==See also==
| |
| {{wikibooks
| |
| | 1 = Econometric Theory
| |
| | 2 = Asymptotic Convergence
| |
| | 3 = Convergence of random variables
| |
| }}
| |
| * [[Proofs of convergence of random variables]]
| |
| * [[Convergence of measures]]
| |
| * [[Continuous stochastic process]]: the question of continuity of a [[stochastic process]] is essentially a question of convergence, and many of the same concepts and relationships used above apply to the continuity question.
| |
| * [[Asymptotic distribution]]
| |
| * [[Big O in probability notation]]
| |
| * [[Skorokhod's representation theorem]]
| |
| * [[Tweedie distributions|The Tweedie convergence theorem]]
| |
| | |
| == Notes ==
| |
| {{reflist}}
| |
| | |
| == References ==
| |
| {{refbegin}}
| |
| * {{cite book|ref=harv
| |
| | last = Bickel
| |
| | first = Peter J.
| |
| | coauthors = Klaassen, Chris A.J.; Ritov, Ya’acov; Wellner, Jon A.
| |
| | year = 1998
| |
| | title = Efficient and adaptive estimation for semiparametric models
| |
| | publisher = Springer-Verlag
| |
| | location = New York
| |
| | isbn = 0-387-98473-9
| |
| | lccn = QA276.8.E374
| |
| | ref = CITEREFBickelKlaassenRitovWellner1998
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Billingsley
| |
| | first = Patrick
| |
| | title = Probability and Measure
| |
| | year = 1986
| |
| | edition = 2nd
| |
| | series = Wiley Series in Probability and Mathematical Statistics
| |
| | publisher = Wiley
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Billingsley
| |
| | first = Patrick
| |
| | year = 1999
| |
| | title = Convergence of probability measures
| |
| | publisher = John Wiley & Sons
| |
| | edition = 2nd
| |
| | pages = 1–28
| |
| | isbn = 0-471-19745-9
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Dudley
| |
| | first = R.M.
| |
| | year = 2002
| |
| | title = Real analysis and probability
| |
| | publisher = Cambridge University Press
| |
| | location = Cambridge, UK
| |
| | isbn = 0-521-80972-X
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | first = G.R.
| |
| | last = Grimmett
| |
| | coauthors = Stirzaker, D.R.
| |
| | year = 1992
| |
| | title = Probability and random processes
| |
| | edition = 2nd
| |
| | publisher = Clarendon Press, Oxford
| |
| | pages = 271–285
| |
| | isbn = 0-19-853665-8
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | first = M.
| |
| | last = Jacobsen
| |
| | year = 1992
| |
| | title = Videregående Sandsynlighedsregning (Advanced Probability Theory)
| |
| | edition = 3rd
| |
| | publisher = HCØ-tryk, Copenhagen
| |
| | pages = 18–20
| |
| | isbn = 87-91180-71-6
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last1 = Ledoux
| |
| | first1 = Michel
| |
| | last2 = Talagrand | first2 = Michel | author2-link = Michel Talagrand
| |
| | title = Probability in Banach spaces
| |
| | publisher = Springer-Verlag
| |
| | location = Berlin
| |
| | year = 1991
| |
| | pages = xii+480
| |
| | isbn = 3-540-52013-9
| |
| | mr = 1102015
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Romano
| |
| | first = Joseph P.
| |
| | coauthor = Siegel, Andrew F.
| |
| | year = 1985
| |
| | title = Counterexamples in probability and statistics
| |
| | publisher = Chapman & Hall
| |
| | location = Great Britain
| |
| | isbn = 0-412-98901-8
| |
| | lccn = QA273.R58 1985
| |
| | ref = CITEREFRomanoSiegel1985
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = van der Vaart
| |
| | first = Aad W.
| |
| | coauthor = Wellner, Jon A.
| |
| | year = 1996
| |
| | title = Weak convergence and empirical processes
| |
| | publisher = Springer-Verlag
| |
| | location = New York
| |
| | isbn = 0-387-94640-3
| |
| | lccn = QA274.V33 1996
| |
| | ref = CITEREFvan_der_VaartWellner1996
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = van der Vaart
| |
| | first = Aad W.
| |
| | title = Asymptotic statistics
| |
| | year = 1998
| |
| | publisher = Cambridge University Press
| |
| | location = New York
| |
| | isbn = 978-0-521-49603-2
| |
| | lccn = QA276.V22 1998
| |
| | ref = CITEREFvan_der_Vaart1998
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Williams
| |
| | first = D.
| |
| | title = Probability with Martingales
| |
| | publisher = Cambridge University Press
| |
| | year = 1991
| |
| | isbn = 0-521-40605-6
| |
| }}
| |
| * {{cite book|ref=harv
| |
| | last = Wong
| |
| | first = E.
| |
| | coauthors = Hájek, B.
| |
| | title = Stochastic Processes in Engineering Systems
| |
| | publisher = Springer–Verlag
| |
| | location = New York
| |
| | year = 1985
| |
| }}
| |
| {{refend}}
| |
| | |
| {{Citizendium|title=Stochastic convergence}}
| |
| | |
| {{DEFAULTSORT:Convergence Of Random Variables}}
| |
| [[Category:Probability theory]]
| |
| [[Category:Statistical theory]]
| |
| [[Category:Stochastic processes]]
| |
| [[Category:Convergence (mathematics)]]
| |