Dunford–Schwartz theorem

In statistics, leverage is a term used in connection with regression analysis and, in particular, in analyses aimed at identifying those observations that are far away from corresponding average predictor values. Leverage points do not necessarily have a large effect on the outcome of fitting regression models.

Leverage points are those observations, if any, made at extreme or outlying values of the independent variables such that the lack of neighboring observations means that the fitted regression model will pass close to that particular observation.

Modern computer packages for statistical analysis include, as part of their facilities for regression analysis, various quantitative measures for identifying influential observations: among these measures is partial leverage, a measure of how a variable contributes to the leverage of a datum.

Definition

The leverage score for the $i^{th}$ data unit is defined as:

• $h_{ii}=(H{)}_{ii}$,

the $i^{th}$ diagonal of the hat matrix $H=X(X^{\prime }X{)}^{-1}{X}^{\prime }$.

Properties

$0\le h_{ii}\le 1$

Proof

First, note that $H^2=X(X^{\prime }X{)}^{-1}{X}^{\prime }X(X^{\prime }X{)}^{-1}{X}^{\prime }=XI(X^{\prime }X{)}^{-1}{X}^{\prime }=H$. Also, observe that $H$ is symmetric. So we have,

• $h_{ii}=h_{ii}^2+\sum _{i\ne j}{h}_{ij}^2\ge 0$

and

• $h_{ii}\ge h_{ii}^2⟹h_{ii}\le 1$

If we are in an ordinary least squares setting with fixed X and:

• $Y=X\beta +ϵ$
• $var(ϵ)={\sigma }^{2}I$

then $var(e_i)=(1-h_{ii}){\sigma }^2$ where $e_i=Y_i-{\hat{Y}}_i$.

In other words, if the $ϵ$ are homoscedastic, leverage scores determine the noise level in the model.

Proof

First, note that $I-H$ is idempotent and symmetric. This gives, $var(e)=var((I-H)Y)=(I-H)var(Y)(I-H{)}^{\prime }={\sigma }^2(I-H{)}^2={\sigma }^2(I-H)$.

So that, $var(e_i)=(1-h_{ii}){\sigma }^2$.

See also

• Hat matrix — whose main diagonal entries are the leverages of the observations
• Mahalanobis distance — a measure of leverage of a datum
• Cook's distance - a measure of changes in regression coefficients when an observation is deleted
• DFFITS
• Outliers — observations with extreme Y values

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.