|
|
| Line 1: |
Line 1: |
| {{Regression bar}}
| | Hello and welcome. My name is Ling. For years she's been residing in Kansas. Bookkeeping is what she does. What he truly enjoys performing is to perform handball but he is struggling to find time for it.<br><br>Review my web blog; [http://makemoneyonline.net.in/blogs/post/9758 makemoneyonline.net.in] |
| The method of '''iteratively reweighted least squares''' ('''IRLS''') is used to solve certain optimization problems. It solves [[objective function]]s of the form:
| |
| | |
| :<math>\underset{\boldsymbol\beta} {\operatorname{arg\,min}} \sum_{i=1}^n w_i (\boldsymbol\beta) \big| y_i - f_i (\boldsymbol\beta) \big|^2, </math>
| |
| | |
| by an [[iterative method]] in which each step involves solving a [[weighted least squares]] problem of the form:
| |
| | |
| :<math>\boldsymbol\beta^{(t+1)} = \underset{\boldsymbol\beta} {\operatorname{arg\,min}} \sum_{i=1}^n w_i (\boldsymbol\beta^{(t)}) \big| y_i - f_i (\boldsymbol\beta) \big|^2. </math>
| |
| | |
| IRLS is used to find the [[maximum likelihood]] estimates of a [[generalized linear model]], and in [[robust regression]] to find an [[M-estimator]], as a way of mitigating the influence of outliers in an otherwise normally-distributed data set. For example, by minimizing the least absolute error rather than the least square error.
| |
| | |
| Although not a linear regression problem, [[Weiszfeld's algorithm]] for approximating the [[geometric median]] can also be viewed as a special case of iteratively reweighted least squares, in which the objective function is the sum of distances of the estimator from the samples.
| |
| | |
| One of the advantages of IRLS over [[linear programming|linear]] and [[convex programming]] is that it can be used with [[Gauss–Newton]] and [[Levenberg–Marquardt]] numerical algorithms.
| |
| | |
| == Examples ==
| |
| | |
| === ''L''<sub>1</sub> minimization for sparse recovery ===
| |
| IRLS can be used for '''[[L1 norm|<math>\ell</math><sub>1</sub>]]''' minimization and smoothed '''[[Lp quasi-norm|<math>\ell</math><sub>p</sub>]]''' minimization, ''p'' < 1, in the [[compressed sensing]] problems. It has been proved that the algorithm has a linear rate of convergence for '''<math>\ell</math><sub>1</sub>''' norm and superlinear for '''<math>\ell</math><sub> ''t''</sub>''' with ''t'' < 1, under the [[restricted isometry property]], which is generally a sufficient condition for sparse solutions.<ref>{{Cite conference
| |
| | last1 = Chartrand | first1 = R.
| |
| | last2 = Yin | first2 = W.
| |
| | title = Iteratively reweighted algorithms for compressive sensing
| |
| | booktitle = IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2008
| |
| | pages = 3869–3872
| |
| | date = March 31 – April 4, 2008
| |
| | url = http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4518498}}
| |
| </ref><ref>{{cite doi|10.1002/cpa.20303}}</ref> In most practical situations, the restricted isometry property is not satisfied.
| |
| | |
| === ''L<sup>p</sup>'' norm linear regression ===
| |
| To find the parameters '''''β''''' = (''β''<sub>1</sub>, …,''β''<sub>''k''</sub>)<sup>T</sup> which minimize the [[Lp space|''L<sup>p</sup>'' norm]] for the [[linear regression]] problem,
| |
| | |
| :<math>
| |
| \underset{\boldsymbol \beta}{ \operatorname{arg\,min} }
| |
| \big\| \mathbf y - X \boldsymbol \beta \|_p
| |
| =
| |
| \underset{\boldsymbol \beta}{ \operatorname{arg\,min} }
| |
| \sum_{i=1}^n \left| y_i - X_i \boldsymbol\beta \right|^p ,
| |
| </math>
| |
| | |
| the IRLS algorithm at step ''t''+1 involves solving the [[Linear least squares (mathematics)#Weighted linear least squares|weighted linear least squares]] problem:<ref>{{cite book
| |
| |chapter=6.8.1 Solutions that Minimize Other Norms of the Residuals
| |
| |title=Matrix algebra
| |
| |last=Gentle |first=James
| |
| |isbn=978-0-387-70872-0
| |
| |doi=10.1007/978-0-387-70873-7
| |
| |publisher=Springer |location=New York
| |
| |year=2007
| |
| }}</ref>
| |
| | |
| :<math>
| |
| \boldsymbol\beta^{(t+1)}
| |
| =
| |
| \underset{\boldsymbol\beta}{ \operatorname{arg\,min} }
| |
| \sum_{i=1}^n w_i^{(t)} \left| y_i - X_i \boldsymbol\beta \right|^2
| |
| =
| |
| (X^{\rm T} W^{(t)} X)^{-1} X^{\rm T} W^{(t)} \mathbf{y},
| |
| </math>
| |
| | |
| where ''W''<sup>(''t'')</sup> is the [[diagonal matrix]] of weights, usually with all elements set initially to:
| |
| | |
| :<math>w_i^{(0)} = 1</math>
| |
| | |
| and updated after each iteration to:
| |
| | |
| :<math>w_i^{(t)} = \big|y_i - X_i \boldsymbol \beta ^{(t)} \big|^{p-2}.</math>
| |
| | |
| In the case ''p'' = 1, this corresponds to [[least absolute deviation]] regression (in this case, the problem would be better approached by use of [[linear programming]] methods,<ref name=Pfeil>William A. Pfeil,
| |
| ''[http://www.wpi.edu/Pubs/E-project/Available/E-project-050506-091720/unrestricted/IQP_Final_Report.pdf Statistical Teaching Aids]'', Bachelor of Science thesis, [[Worcester Polytechnic Institute]], 2006</ref> so the result would be exact) and the formula is:
| |
| | |
| :<math>w_i^{(t)} = \frac{1}{\big|y_i - X_i \boldsymbol \beta ^{(t)} \big|}.</math>
| |
| | |
| To avoid dividing by zero, [[Regularization (mathematics)|regularization]] must be done, so in practice the formula is:
| |
| | |
| :<math>w_i^{(t)} = \frac{1}{\text{max}(\delta, \big|y_i - X_i \boldsymbol \beta ^{(t)} \big|)}.</math>
| |
| | |
| where <math>\delta</math> is some small value, like 0.0001.<ref name=Pfeil />
| |
| | |
| == Notes ==
| |
| {{Reflist}}
| |
| | |
| == References ==
| |
| * [http://amath.colorado.edu/courses/7400/2010Spr/lecture23.pdf University of Colorado Applied Regression lecture slides]
| |
| * [http://sepwww.stanford.edu/public/docs/sep103/antoine2/paper_html/index.html Stanford Lecture Notes on the IRLS algorithm by Antoine Guitton]
| |
| * [http://www.mai.liu.se/~akbjo/LSPbook.html Numerical Methods for Least Squares Problems by Åke Björck] (Chapter 4: Generalized Least Squares Problems.)
| |
| * [http://graphics.stanford.edu/~jplewis/lscourse/SLIDES.pdf Practical Least-Squares for Computer Graphics. SIGGRAPH Course 11]
| |
| | |
| {{DEFAULTSORT:Iteratively Reweighted Least Squares}}
| |
| [[Category:Regression analysis]]
| |
| [[Category:Least squares]]
| |
Hello and welcome. My name is Ling. For years she's been residing in Kansas. Bookkeeping is what she does. What he truly enjoys performing is to perform handball but he is struggling to find time for it.
Review my web blog; makemoneyonline.net.in