Witt's theorem: Difference between revisions

Revision as of 21:12, 26 April 2013

In statistics, M-estimators are a broad class of estimators, which are obtained as the minima of sums of functions of the data. Least-squares estimators are M-estimators. The definition of M-estimators was motivated by robust statistics, which contributed new types of M-estimators. The statistical procedure of evaluating an M-estimator on a data set is called M-estimation.

More generally, an M-estimator may be defined to be a zero of an estimating function.^[1]^[2]^[3]^[4]^[5]^[6] This estimating function is often the derivative of another statistical function: For example, a maximum-likelihood estimate is often defined to be a zero of the derivative of the likelihood function with respect to the parameter: thus, a maximum-likelihood estimator is often a critical point of the score function.^[7] In many applications, such M-estimators can be thought of as estimating characteristics of the population.

Historical motivation

The method of least squares is a prototypical M-estimator, since the estimator is defined as a minimum of the sum of squares of the residuals.

Another popular M-estimator is maximum-likelihood estimation. For a family of probability density functions f parameterized by θ, a maximum likelihood estimator of θ is computed for each set of data by maximizing the likelihood function over the parameter space { θ } . When the observations are independent and identically distributed, a ML-estimate $\hat{θ}$ satisfies

\hat{θ} = \arg \max_{θ} (\prod_{i = 1}^{n} f (x_{i}, θ))

or, equivalently,

\hat{θ} = \arg \min_{θ} (- \sum_{i = 1}^{n} \log (f (x_{i}, θ))) .

Maximum-likelihood estimators are often inefficient and biased for finite samples. For many regular problems, maximum-likelihood estimation performs well for "large samples", being an approximation of a posterior mode. If the problem is "regular", then any bias of the MLE (or posterior mode) decreases to zero when the sample-size increases to infinity. The performance of maximum-likelihood (and posterior-mode) estimators drops when the parametric family is mis-specified.

Definition

In 1964, Peter Huber proposed generalizing maximum likelihood estimation to the minimization of

\sum_{i = 1}^{n} ρ (x_{i}, θ),

where ρ is a function with certain properties (see below). The solutions

\hat{θ} = \arg \min_{θ} (\sum_{i = 1}^{n} ρ (x_{i}, θ))

are called M-estimators ("M" for "maximum likelihood-type" (Huber, 1981, page 43)); other types of robust estimator include L-estimators, R-estimators and S-estimators. Maximum likelihood estimators (MLE) are thus a special case of M-estimators. With suitable rescaling, M-estimators are special cases of extremum estimators (in which more general functions of the observations can be used).

The function ρ, or its derivative, ψ, can be chosen in such a way to provide the estimator desirable properties (in terms of bias and efficiency) when the data are truly from the assumed distribution, and 'not bad' behaviour when the data are generated from a model that is, in some sense, close to the assumed distribution.

Types of M-estimators

M-estimators are solutions, θ, which minimize

\sum_{i = 1}^{n} ρ (x_{i}, θ) .

This minimization can always be done directly. Often it is simpler to differentiate with respect to θ and solve for the root of the derivative. When this differentiation is possible, the M-estimator is said to be of ψ-type. Otherwise, the M-estimator is said to be of ρ-type.

In most practical cases, the M-estimators are of ψ-type.

ρ-type

For positive integer r, let $(𝒳, Σ)$ and $(Θ \subset ℝ^{r}, S)$ be measure spaces. $θ \in Θ$ is a vector of parameters. An M-estimator of ρ-type T is defined through a measurable function $ρ : 𝒳 \times Θ \to ℝ$ . It maps a probability distribution F on $𝒳$ to the value $T (F) \in Θ$ (if it exists) that minimizes $\int_{𝒳} ρ (x, θ) d F (x)$ :

T (F) : = \arg \min_{θ \in Θ} \int_{𝒳} ρ (x, θ) d F (x)

For example, for the maximum likelihood estimator, $ρ (x, θ) = - \log (f (x, θ))$ , where $f (x, θ) = \frac{\partial F (x, θ)}{\partial x}$ .

ψ-type

If $ρ$ is differentiable, the computation of $\hat{θ}$ is usually much easier. An M-estimator of ψ-type T is defined through a measurable function $ψ : 𝒳 \times Θ \to ℝ^{r}$ . It maps a probability distribution F on $𝒳$ to the value $T (F) \in Θ$ (if it exists) that solves the vector equation:

\int_{𝒳} ψ (x, θ) d F (x) = 0

\int_{𝒳} ψ (x, T (F)) d F (x) = 0

For example, for the maximum likelihood estimator, $ψ (x, θ) = {(\frac{\partial \log (f (x, θ))}{\partial θ^{1}}, \dots, \frac{\partial \log (f (x, θ))}{\partial θ^{p}})}^{T}$ , where $u^{T}$ denotes the transpose of vector u and $f (x, θ) = \frac{\partial F (x, θ)}{\partial x}$ .

Such an estimator is not necessarily an M-estimator of ρ-type, but if ρ has a continuous first derivative with respect to $θ$ , then a necessary corresponding M-estimator of ψ-type to be an M-estimator of ρ-type is $ψ (x, θ) = \nabla_{θ} ρ (x, θ)$ . The previous definitions can easily be extended to finite samples.

If the function ψ decreases to zero as $x \to \pm \infty$ , the estimator is called redescending. Such estimators have some additional desirable properties, such as complete rejection of gross outliers.

Computation

For many choices of ρ or ψ, no closed form solution exists and an iterative approach to computation is required. It is possible to use standard function optimization algorithms, such as Newton-Raphson. However, in most cases an iteratively re-weighted least squares fitting algorithm can be performed; this is typically the preferred method.

For some choices of ψ, specifically, redescending functions, the solution may not be unique. The issue is particularly relevant in multivariate and regression problems. Thus, some care is needed to ensure that good starting points are chosen. Robust starting points, such as the median as an estimate of location and the median absolute deviation as a univariate estimate of scale, are common.

Properties

Distribution

It can be shown that M-estimators are asymptotically normally distributed. As such, Wald-type approaches to constructing confidence intervals and hypothesis tests can be used. However, since the theory is asymptotic, it will frequently be sensible to check the distribution, perhaps by examining the permutation or bootstrap distribution.

Influence function

The influence function of an M-estimator of $ψ$ -type is proportional to its defining $ψ$ function.

Let T be an M-estimator of ψ-type, and G be a probability distribution for which $T (G)$ is defined. Its influence function IF is

IF (x; T, G) = - \frac{ψ (x, T (G))}{\int [\frac{\partial ψ (y, θ)}{\partial θ}] d y}

A proof of this property of M-estimators can be found in Huber (1981, Section 3.2).

Applications

M-estimators can be constructed for location parameters and scale parameters in univariate and multivariate settings, as well as being used in robust regression.

Examples

Mean

Let (X₁, ..., X_n) be a set of independent, identically distributed random variables, with distribution F.

If we define

ρ (x, θ) = \frac{(x - θ)^{2}}{2},

we note that this is minimized when θ is the mean of the Xs. Thus the mean is an M-estimator of ρ-type, with this ρ function.

As this ρ function is continuously differentiable in θ, the mean is thus also an M-estimator of ψ-type for ψ(x, θ) = θ − x.

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

External links

M-estimators — an introduction to the subject by Zhengyou Zhang
M-estimators - an interactive demonstration of Huber's M-estimator

↑ V. P. Godambe, editor. Estimating functions, volume 7 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 1991.
↑ Christopher C. Heyde. Quasi-likelihood and its application: A general approach to optimal parameter estimation. Springer Series in Statistics. Springer-Verlag, New York, 1997.
↑ D. L. McLeish and Christopher G. Small. The theory and applications of statistical inference functions, volume 44 of Lecture Notes in Statistics. Springer-Verlag, New York, 1988.
↑ Parimal Mukhopadhyay. An Introduction to Estimating Functions. Alpha Science International, Ltd, 2004.
↑ Christopher G. Small and Jinfang Wang. Numerical methods for nonlinear estimating equations, volume 29 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 2003.
↑ Sara A. van de Geer. Empirical Processes in M-estimation: Applications of empirical process theory, volume 6 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2000.
↑ One of the biggest reasons investing in a Singapore new launch is an effective things is as a result of it is doable to be lent massive quantities of money at very low interest rates that you should utilize to purchase it. Then, if property values continue to go up, then you'll get a really high return on funding (ROI). Simply make sure you purchase one of the higher properties, reminiscent of the ones at Fernvale the Riverbank or any Singapore landed property Get Earnings by means of Renting

In its statement, the singapore property listing - website link, government claimed that the majority citizens buying their first residence won't be hurt by the new measures. Some concessions can even be prolonged to chose teams of consumers, similar to married couples with a minimum of one Singaporean partner who are purchasing their second property so long as they intend to promote their first residential property. Lower the LTV limit on housing loans granted by monetary establishments regulated by MAS from 70% to 60% for property purchasers who are individuals with a number of outstanding housing loans on the time of the brand new housing purchase. Singapore Property Measures - 30 August 2010 The most popular seek for the number of bedrooms in Singapore is 4, followed by 2 and three. Lush Acres EC @ Sengkang

Discover out more about real estate funding in the area, together with info on international funding incentives and property possession. Many Singaporeans have been investing in property across the causeway in recent years, attracted by comparatively low prices. However, those who need to exit their investments quickly are likely to face significant challenges when trying to sell their property – and could finally be stuck with a property they can't sell. Career improvement programmes, in-house valuation, auctions and administrative help, venture advertising and marketing, skilled talks and traisning are continuously planned for the sales associates to help them obtain better outcomes for his or her shoppers while at Knight Frank Singapore. No change Present Rules

Extending the tax exemption would help. The exemption, which may be as a lot as $2 million per family, covers individuals who negotiate a principal reduction on their existing mortgage, sell their house short (i.e., for lower than the excellent loans), or take part in a foreclosure course of. An extension of theexemption would seem like a common-sense means to assist stabilize the housing market, but the political turmoil around the fiscal-cliff negotiations means widespread sense could not win out. Home Minority Chief Nancy Pelosi (D-Calif.) believes that the mortgage relief provision will be on the table during the grand-cut price talks, in response to communications director Nadeam Elshami. Buying or promoting of blue mild bulbs is unlawful.

A vendor's stamp duty has been launched on industrial property for the primary time, at rates ranging from 5 per cent to 15 per cent. The Authorities might be trying to reassure the market that they aren't in opposition to foreigners and PRs investing in Singapore's property market. They imposed these measures because of extenuating components available in the market." The sale of new dual-key EC models will even be restricted to multi-generational households only. The models have two separate entrances, permitting grandparents, for example, to dwell separately. The vendor's stamp obligation takes effect right this moment and applies to industrial property and plots which might be offered inside three years of the date of buy. JLL named Best Performing Property Brand for second year running

The data offered is for normal info purposes only and isn't supposed to be personalised investment or monetary advice. Motley Fool Singapore contributor Stanley Lim would not personal shares in any corporations talked about. Singapore private home costs increased by 1.eight% within the fourth quarter of 2012, up from 0.6% within the earlier quarter. Resale prices of government-built HDB residences which are usually bought by Singaporeans, elevated by 2.5%, quarter on quarter, the quickest acquire in five quarters. And industrial property, prices are actually double the levels of three years ago. No withholding tax in the event you sell your property. All your local information regarding vital HDB policies, condominium launches, land growth, commercial property and more

There are various methods to go about discovering the precise property. Some local newspapers (together with the Straits Instances ) have categorised property sections and many local property brokers have websites. Now there are some specifics to consider when buying a 'new launch' rental. Intended use of the unit Every sale begins with 10 p.c low cost for finish of season sale; changes to 20 % discount storewide; follows by additional reduction of fiftyand ends with last discount of 70 % or extra. Typically there is even a warehouse sale or transferring out sale with huge mark-down of costs for stock clearance. Deborah Regulation from Expat Realtor shares her property market update, plus prime rental residences and houses at the moment available to lease Esparina EC @ Sengkang

[1] V. P. Godambe, editor. Estimating functions, volume 7 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 1991.

[2] Christopher C. Heyde. Quasi-likelihood and its application: A general approach to optimal parameter estimation. Springer Series in Statistics. Springer-Verlag, New York, 1997.

[3] D. L. McLeish and Christopher G. Small. The theory and applications of statistical inference functions, volume 44 of Lecture Notes in Statistics. Springer-Verlag, New York, 1988.

[4] Parimal Mukhopadhyay. An Introduction to Estimating Functions. Alpha Science International, Ltd, 2004.

[5] Christopher G. Small and Jinfang Wang. Numerical methods for nonlinear estimating equations, volume 29 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 2003.

[6] Sara A. van de Geer. Empirical Processes in M-estimation: Applications of empirical process theory, volume 6 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2000.

[7] One of the biggest reasons investing in a Singapore new launch is an effective things is as a result of it is doable to be lent massive quantities of money at very low interest rates that you should utilize to purchase it. Then, if property values continue to go up, then you'll get a really high return on funding (ROI). Simply make sure you purchase one of the higher properties, reminiscent of the ones at Fernvale the Riverbank or any Singapore landed property Get Earnings by means of Renting

In its statement, the singapore property listing - website link, government claimed that the majority citizens buying their first residence won't be hurt by the new measures. Some concessions can even be prolonged to chose teams of consumers, similar to married couples with a minimum of one Singaporean partner who are purchasing their second property so long as they intend to promote their first residential property. Lower the LTV limit on housing loans granted by monetary establishments regulated by MAS from 70% to 60% for property purchasers who are individuals with a number of outstanding housing loans on the time of the brand new housing purchase. Singapore Property Measures - 30 August 2010 The most popular seek for the number of bedrooms in Singapore is 4, followed by 2 and three. Lush Acres EC @ Sengkang

Discover out more about real estate funding in the area, together with info on international funding incentives and property possession. Many Singaporeans have been investing in property across the causeway in recent years, attracted by comparatively low prices. However, those who need to exit their investments quickly are likely to face significant challenges when trying to sell their property – and could finally be stuck with a property they can't sell. Career improvement programmes, in-house valuation, auctions and administrative help, venture advertising and marketing, skilled talks and traisning are continuously planned for the sales associates to help them obtain better outcomes for his or her shoppers while at Knight Frank Singapore. No change Present Rules

Extending the tax exemption would help. The exemption, which may be as a lot as $2 million per family, covers individuals who negotiate a principal reduction on their existing mortgage, sell their house short (i.e., for lower than the excellent loans), or take part in a foreclosure course of. An extension of theexemption would seem like a common-sense means to assist stabilize the housing market, but the political turmoil around the fiscal-cliff negotiations means widespread sense could not win out. Home Minority Chief Nancy Pelosi (D-Calif.) believes that the mortgage relief provision will be on the table during the grand-cut price talks, in response to communications director Nadeam Elshami. Buying or promoting of blue mild bulbs is unlawful.

A vendor's stamp duty has been launched on industrial property for the primary time, at rates ranging from 5 per cent to 15 per cent. The Authorities might be trying to reassure the market that they aren't in opposition to foreigners and PRs investing in Singapore's property market. They imposed these measures because of extenuating components available in the market." The sale of new dual-key EC models will even be restricted to multi-generational households only. The models have two separate entrances, permitting grandparents, for example, to dwell separately. The vendor's stamp obligation takes effect right this moment and applies to industrial property and plots which might be offered inside three years of the date of buy. JLL named Best Performing Property Brand for second year running

The data offered is for normal info purposes only and isn't supposed to be personalised investment or monetary advice. Motley Fool Singapore contributor Stanley Lim would not personal shares in any corporations talked about. Singapore private home costs increased by 1.eight% within the fourth quarter of 2012, up from 0.6% within the earlier quarter. Resale prices of government-built HDB residences which are usually bought by Singaporeans, elevated by 2.5%, quarter on quarter, the quickest acquire in five quarters. And industrial property, prices are actually double the levels of three years ago. No withholding tax in the event you sell your property. All your local information regarding vital HDB policies, condominium launches, land growth, commercial property and more

There are various methods to go about discovering the precise property. Some local newspapers (together with the Straits Instances ) have categorised property sections and many local property brokers have websites. Now there are some specifics to consider when buying a 'new launch' rental. Intended use of the unit Every sale begins with 10 p.c low cost for finish of season sale; changes to 20 % discount storewide; follows by additional reduction of fiftyand ends with last discount of 70 % or extra. Typically there is even a warehouse sale or transferring out sale with huge mark-down of costs for stock clearance. Deborah Regulation from Expat Realtor shares her property market update, plus prime rental residences and houses at the moment available to lease Esparina EC @ Sengkang

[1]

[2]

[3]

[4]

[5]

[6]

[7]

@@ Line 1: / Line 1: @@
-Greetings. The writer's name is Phebe and she feels comfy when people use the full title. Doing ceramics is what my family and I appreciate. He used to be unemployed but now he is a computer operator but his marketing never [http://Www.Uic.edu/classes/bios/bios104/mike/bacteria01.htm arrives]. North Dakota is her beginning  home std test  [http://www.articlestunner.com/cures-for-the-yeast-infection-suggestions-to-use-now/ at home std testing] location but she will have to move one day or an  over the counter std test additional.<br><br>Here is my blog post - at home  [http://withlk.com/board_RNsI08/10421 std home test] std test ([http://bikedance.com/blogs/post/15246 please click the following web site])
+In [[statistics]], '''M-estimators''' are a broad class of [[estimator]]s, which are obtained as the minima of sums of functions of the data. Least-squares estimators are M-estimators. The definition of M-estimators was motivated by [[robust statistics]], which contributed new types of M-estimators. The statistical procedure of evaluating an M-estimator on a data set is called '''M-estimation'''.
+More generally, an M-estimator may be defined to be a zero of an [[estimating equations|estimating function]].<ref>V. P. Godambe, editor. ''Estimating functions'', volume 7 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 1991.</ref><ref>Christopher C. Heyde. ''Quasi-likelihood and its application: A general approach to optimal parameter estimation''. Springer Series in Statistics. Springer-Verlag, New York, 1997.</ref><ref>D. L. McLeish and Christopher G. Small. ''The theory and applications of statistical inference functions'', volume 44 of Lecture Notes in Statistics. Springer-Verlag, New York, 1988.</ref><ref>Parimal Mukhopadhyay. ''An Introduction to Estimating Functions''. Alpha Science International, Ltd, 2004.</ref><ref>Christopher G. Small and Jinfang Wang. ''Numerical methods for nonlinear estimating equations'', volume 29 of Oxford Statistical Science Series. The Clarendon Press Oxford University Press, New York, 2003.</ref><ref>Sara A. van de Geer. ''Empirical Processes in M-estimation: Applications of empirical process theory,'' volume 6 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2000.</ref> This estimating function is often the derivative of another statistical function: For example, a maximum-likelihood estimate is often defined to be a zero of the derivative of the likelihood function with respect to the parameter: thus, a maximum-likelihood estimator is often a [[Critical_point_(mathematics)|critical point]] of the [[Score (statistics)|score]] function.<ref>{{cite journal | last = Ferguson | first = Thomas&nbsp;S. | title = An inconsistent maximum likelihood estimate | journal = Journal of the American Statistical Association | volume = 77 | issue = 380 | year = 1982 | pages = 831–834 | jstor=2287314 }}</ref>  In many applications, such M-estimators can be thought of as estimating characteristics of the population.
+==Historical motivation==
+The method of [[least squares]] is a prototypical M-estimator, since the estimator is defined as a minimum of the sum of squares of the residuals.
+Another popular M-estimator is maximum-likelihood estimation. For a family of [[probability density function]]s ''f'' parameterized by ''θ'', a [[maximum likelihood]] estimator of ''θ'' is computed for each set of data by maximizing the [[likelihood function]] over the parameter space  {&nbsp;''θ''&nbsp;}&nbsp;.  When the observations are independent and identically distributed, a ML-estimate <math>\hat{\theta}</math> satisfies
+:<math>\widehat{\theta} = \arg\max_{\displaystyle\theta}{ \left( \prod_{i=1}^n f(x_i, \theta) \right) }\,\!</math>
+or, equivalently,
+:<math>\widehat{\theta} = \arg\min_{\displaystyle\theta}{ \left( -\sum_{i=1}^n \log{( f(x_i, \theta) ) }\right) }.\,\!</math>
+Maximum-likelihood estimators are often inefficient and biased for finite samples. For many regular problems, maximum-likelihood estimation performs well for "large samples", being an approximation of a [[MAP estimator|posterior mode]]. If the problem is "regular", then any bias of the MLE (or posterior mode) decreases to zero when the sample-size increases to infinity. The performance of maximum-likelihood (and posterior-mode) estimators drops when the parametric family is mis-specified.
+==Definition==
+In 1964, [[Peter Huber]] proposed generalizing maximum likelihood estimation to the minimization of
+:<math>\sum_{i=1}^n\rho(x_i, \theta),\,\!</math>
+where ρ is a function with certain properties (see below). The solutions
+:<math>\hat{\theta} = \arg\min_{\displaystyle\theta}\left(\sum_{i=1}^n\rho(x_i, \theta)\right) \,\!</math>
+are called '''M-estimators''' ("M" for "maximum likelihood-type" (Huber, 1981, page 43)); other types of robust estimator include [[L-estimator]]s, [[R-estimator]]s and [[S-estimator]]s.  Maximum likelihood estimators (MLE) are thus a special case of M-estimators. With suitable rescaling, M-estimators are special cases of [[extremum estimator]]s (in which more general functions of the observations can be used).
+The function ρ, or its derivative, ψ, can be chosen in such a way to provide the estimator desirable properties (in terms of bias and efficiency) when the data are truly from the assumed distribution, and 'not bad' behaviour when the data are generated from a model that is, in some sense, ''close'' to the assumed distribution.
+==Types of M-estimators==
+M-estimators are solutions, ''θ'', which minimize
+:<math>\sum_{i=1}^n\rho(x_i,\theta).\,\!</math>
+This minimization can always be done directly. Often it is simpler to differentiate with respect to ''θ'' and solve for the root of the derivative. When this differentiation is possible,  the M-estimator is said to be of '''ψ-type'''.  Otherwise, the M-estimator is said to be of '''ρ-type'''.
+In most practical cases, the M-estimators are of ψ-type.
+===ρ-type===
+For positive integer ''r'', let <math>(\mathcal{X},\Sigma)</math> and <math>(\Theta\subset\mathbb{R}^r,S)</math> be measure spaces. <math>\theta\in\Theta</math> is a vector of parameters. An M-estimator of ρ-type ''T'' is defined through a [[measurable function]] <math>\rho:\mathcal{X}\times\Theta\rightarrow\mathbb{R}</math>. It maps a probability distribution ''F'' on <math>\mathcal{X}</math> to the value <math>T(F)\in\Theta</math> (if it exists) that minimizes
+<math>\int_{\mathcal{X}}\rho(x,\theta)dF(x)</math>:
+: <math>T(F):=\arg\min_{\theta\in\Theta}\int_{\mathcal{X}}\rho(x,\theta)dF(x)</math>
+For example, for the [[maximum likelihood]] estimator, <math>\rho(x,\theta)=-\log(f(x,\theta))</math>, where <math>f(x,\theta)=\frac{\partial F(x,\theta)}{\partial x}</math>.
+===ψ-type===
+If <math>\rho</math> is differentiable, the computation of <math>\widehat{\theta}</math> is usually much easier. An M-estimator of ψ-type ''T'' is defined through a measurable function <math>\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^r</math>. It maps a probability distribution ''F'' on <math>\mathcal{X}</math> to the value <math>T(F)\in\Theta</math> (if it exists) that solves the vector equation:
+: <math>\int_{\mathcal{X}}\psi(x,\theta) \, dF(x)=0</math>
+: <math>\int_{\mathcal{X}}\psi(x,T(F)) \, dF(x)=0</math>
+For example, for the [[maximum likelihood]] estimator, <math>\psi(x,\theta)=\left(\frac{\partial\log(f(x,\theta))}{\partial \theta^1},\dots,\frac{\partial\log(f(x,\theta))}{\partial \theta^p}\right)^\mathrm{T}</math>, where <math>u^\mathrm{T}</math> denotes the transpose of vector ''u'' and <math>f(x,\theta)=\frac{\partial F(x,\theta)}{\partial x}</math>.
+Such an estimator is not necessarily an M-estimator of ρ-type, but if ρ has a continuous first derivative with respect to <math>\theta</math>, then a necessary corresponding M-estimator of ψ-type to be an M-estimator of ρ-type is <math>\psi(x,\theta)=\nabla_\theta\rho(x,\theta)</math>. The previous definitions can easily be extended to finite samples.
+If the function ψ decreases to zero as <math>x \rightarrow \pm \infty</math>, the estimator is called [[redescending M-estimator|redescending]]. Such estimators have some additional desirable properties, such as complete rejection of gross outliers.
+==Computation==
+For many choices of ρ or ψ, no closed form solution exists and an iterative approach to computation is required. It is possible to use standard function optimization algorithms, such as Newton-Raphson. However, in most cases an [[iteratively re-weighted least squares]] fitting algorithm can be performed; this is typically the preferred method.
+For some choices of ψ, specifically, ''[[Redescending M-estimator|redescending]]'' functions, the solution may not be unique. The issue is particularly relevant in multivariate and regression problems. Thus, some care is needed to ensure that good starting points are chosen.  [[Robust statistics|Robust]] starting points, such as the [[median]] as an estimate of location and the [[median absolute deviation]] as a univariate estimate of scale, are common.
+==Properties==
+===Distribution===
+It can be shown that M-estimators are asymptotically normally distributed. As such, [[Wald-type test|Wald-type approaches]] to constructing confidence intervals and hypothesis tests can be used. However, since the theory is asymptotic, it will frequently be sensible to check the distribution, perhaps by examining the permutation or [[bootstrap (statistics)|bootstrap]] distribution.
+===Influence function===
+The influence function of an M-estimator of <math>\psi</math>-type is proportional to its defining <math>\psi</math> function.
+Let ''T'' be an M-estimator of ψ-type, and ''G'' be a probability distribution for which <math>T(G)</math> is defined.<!-- and let <math>x\in\mathcal{X}</math>.--> Its influence function IF is
+<!--
+what's the variable of integration in the denominator?
+     Asssuming it is "y"...
+-->
+:<math>\operatorname{IF}(x;T,G) = -\frac{\psi(x,T(G))}
+                                       {\int\left[\frac{\partial\psi(y,\theta)}
+                                                       {\partial\theta}
+                                            \right] \mathrm{d}y
+                                       }
+</math>
+A proof of this property of M-estimators can be found in Huber (1981, Section 3.2).
+<!--
+Proof:
+By definition, <math>\forall G\in\mbox{dom}(T),\int\psi(x,T(G))dG(x)=0</math>.  Let
+<math>c(0)=G</math> and <math>c'(0)=\Delta_x-G</math>, for example <math>c(t)=G+t(\Delta_x-G)</math>. Then
+:<math>\forall t\in\mathcal{X},\int\psi(y,T(c(t)))d(c(t)(y))=0</math>
+Differentiating yields
+:<math>\forall
+t\in\mathcal{X},\frac{\partial}{\partial t}\int\psi(y,T(c(t)))d(c(t)(y))=0</math>
+We know that <math>dc(t)=td(\Delta_x-G)+dF</math>. Therefore,
+:<math>\forall
+t\in\mathcal{X},\frac{\partial}{\partial t}\int\psi(y,T(c(t)))td(\Delta_x-G)(y)+\frac{\partial}{\partial t}\int\psi(x,T(c(t)))dG(y)=0</math>
+Supposing differentiation and integration can be interchanged,
+:<math>\forall
+t\in\mathcal{X},t\int\frac{\partial\psi(y,T(c(t)))}{\partial t}d(\Delta_x-G)(y)+\int\psi(y,T(c(t)))d(\Delta_x-G)(y)+\int\frac{\partial\psi(x,T(c(t)))}{\partial t}dG(x)=0</math>
+As
+<math>\int\psi(y,T(c(t)))d(\Delta_x-G)(y)</math>
+<math>=\int\psi(y,T(c(t)))d(\Delta_x)(y)-\int\psi(y,T(c(t)))dG(y)=\psi(x,T(c(t)))-0</math>
+we can write:
+:<math>\forall
+t\in\mathcal{X}, \psi(x,c(t))+t\int\frac{\partial\psi(y,T(c(t)))}{\partial t}d(\Delta_x-G)(y)+\int\frac{\partial\psi(x,T(c(t)))}{\partial t}dG(x)=0</math>
+Now,
+<math>\frac{\partial\psi(y,T(c(t)))}{\partial t}=\left[\frac{\partial\psi(x,\theta)}{\partial\theta}\right]_{T(c(t))}</math>
+Therefore,
+<math>\forall
+t\in\mathcal{X},\psi(x,c(t))+t\int\frac{\partial\psi(y,T(c(t)))}{\partial t}d(\Delta_x-G)(y)+</math>
+<math>\int\left[\frac{\partial\psi(x,\theta)}{\partial\theta}\right]_{T(c(t))}dG(x)\frac{\partial T(c(t))}{\partial t}=0</math>
+As this equation is valid for all ''t'' in <math>\mathcal{X}</math>, we can take <math>t=0</math>:
+<math>\psi(x,T(G))+\int\left[\frac{\partial\psi(x,\theta)}{\partial\theta}\right]_{T(G)}dG(x)\left[\frac{\partial T(c(t))}{\partial t}\right]_{t=0}=0</math>
+By definition,
+<math>\left[\frac{\partial T(c(t))}{\partial t}\right]_{t=0}=d_GT(\Delta_x-G)=IF(x;T,G)</math>, hence
+<math>IF(x;T,G)=-\frac{\psi(x,T(G))}{\int\left[\frac{\partial\psi(y,\theta)}{\partial\theta}\right]}</math>
+which completes the proof.-->
+==Applications==
+M-estimators can be constructed for location parameters and scale parameters in univariate and multivariate settings, as well as being used in robust regression.
+==Examples==
+===Mean===
+Let (''X''<sub>1</sub>, ..., ''X''<sub>''n''</sub>) be a set of [[iid|independent, identically distributed]] random variables, with distribution ''F''.
+If we define
+:<math>\rho(x, \theta)=\frac{(x - \theta)^2}{2},\,\!</math>
+we note that this is minimized when ''θ'' is the [[arithmetic mean|mean]] of the ''X''s.  Thus the mean is an M-estimator of ρ-type, with this ρ function.
+As this ρ function is continuously differentiable in ''θ'', the mean is thus also an M-estimator of ψ-type for ψ(''x'', ''θ'') = ''θ''&nbsp;−&nbsp;''x''.
+==See also==
+*[[Robust statistics]]
+*[[Robust regression]]
+*[[Redescending M-estimator]]
+==References==
+{{Reflist}}
+==Further reading==
+* {{cite book  | last = Andersen | first = Robert | title = Modern Methods for Robust Regression | publisher = Sage Publications | location = Los Angeles, CA | series =  Quantitative Applications in the Social Sciences | volume = 152| year = 2008 | isbn = 978-1-4129-4072-6}}
+* {{cite book  | last = Godambe  | first = V. P. | title = Estimating functions | publisher = Clarendon Press | location=New York | series= Oxford Statistical Science Series | volume = 7 | year = 1991 | isbn = 978-0-19-852228-7 }}
+* {{cite book | last = Heyde | first = Christopher C. | title = Quasi-likelihood and its application: A general approach to optimal parameter estimation  | publisher = Springer | location=New York | year = 1997 | series= Springer Series in Statistics | isbn = 978-0-387-98225-0 | doi = 10.1007/b98823 }}
+* {{cite book | last = Huber | first = Peter J. | title = Robust Statistics | edition= 2nd | publisher = John Wiley & Sons Inc. | location = Hoboken, NJ | year = 2009 | isbn = 978-0-470-12990-6 }}
+* {{cite book  | last = Hoaglin | first = David C. | coauthors = Frederick Mosteller and John W. Tukey | title = Understanding Robust and Exploratory Data Analysis | publisher = John Wiley & Sons Inc. | location = Hoboken, NJ | year = 1983 | isbn = 0-471-09777-2 }}
+* {{cite book  | last = McLeish | first = D.L. | coauthors = Christopher G. Small | title = The theory and applications of statistical inference functions | publisher = Springer | location=New York | year = 1989 | series= Lecture Notes in Statistics | volume = 44 | isbn = 978-0-387-96720-2 }}
+* {{cite book | last = Mukhopadhyay | first = Parimal  | title = An Introduction to Estimating Functions | publisher = Alpha Science International, Ltd | location = Harrow, UK | year = 2004 | isbn = 978-1-84265-163-6 }}
+*{{Citation | last1=Press | first1=WH | last2=Teukolsky | first2=SA | last3=Vetterling | first3=WT | last4=Flannery | first4=BP | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press |  publication-place=New York | isbn=978-0-521-88068-8 | chapter=Section 15.7. Robust Estimation | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=818}}
+* {{cite book  | last = Serfling  | first = Robert J. | title = Approximation theorems of mathematical statistics | publisher = John Wiley & Sons Inc. | location = Hoboken, NJ | year = 2002 | series = Wiley Series in Probability and Mathematical Statistics | isbn = 978-0-471-21927-9 }}
+* {{cite journal | last=Shapiro  | first=Alexander | title=On the asymptotics of constrained local ''M''-estimators | journal= Annals of Statistics | volume=28 | issue=3 | year=2000 |pages=948–960 | doi=10.1214/aos/1015952006  | mr=1792795 | jstor = 2674061}}
+* {{cite book  | last = Small | first = Christopher G. | coauthors = Jinfang Wang | title = Numerical methods for nonlinear estimating equations | publisher = Oxford University Press | location = New York | year = 2003 | series= Oxford Statistical Science Series | volume = 29 | isbn = 978-0-19-850688-1 }}
+* {{cite book | last = van de Geer | first = Sara A. | title = Empirical Processes in M-estimation: Applications of empirical process theory | publisher = Cambridge University Press | location = Cambridge, UK | year = 2000 | series = Cambridge Series in Statistical and Probabilistic Mathematics | volume = 6 | isbn = 978-0-521-65002-1 | doi = 10.2277/052165002X }}
+* {{cite book | last = Wilcox | first = R. R. | title = Applying contemporary statistical techniques <!-- I think "Summarizing data" is the chapter name; no need since pages give --> | publisher = San Diego, CA: Academic Press | year = 2003 | pages = 55–79 }}
+* {{cite book | last = Wilcox | first = R. R. | title = Introduction to Robust Estimation and Hypothesis Testing, 3rd Ed | publisher = San Diego, CA: Academic Press | year = 2012 }}
+==External links==
+* [http://research.microsoft.com/en-us/um/people/zhang/INRIA/Publis/Tutorial-Estim/node24.html#SECTION000104000000000000000 M-estimators] — an introduction to the subject by Zhengyou Zhang
+* [http://www.dinamistics.com/post/2012/M-estimators/ M-estimators] - an interactive demonstration of Huber's M-estimator
+{{DEFAULTSORT:M-Estimator}}
+[[Category:M-estimators]]
+[[Category:Estimation theory]]
+[[Category:Robust regression]]
+[[Category:Robust statistics]]

Witt's theorem: Difference between revisions

Revision as of 21:12, 26 April 2013

Contents

Historical motivation

Definition

Types of M-estimators

ρ-type

ψ-type

Computation

Properties

Distribution

Influence function

Applications

Examples

Mean

See also

References

Further reading

External links

Navigation menu

Witt's theorem: Difference between revisions

Revision as of 21:12, 26 April 2013

Historical motivation

Definition

Types of M-estimators

ρ-type

ψ-type

Computation

Properties

Distribution

Influence function

Applications

Examples

Mean

See also

References

Further reading

External links

Navigation menu

Search