|
|
| Line 1: |
Line 1: |
| In [[mathematics]] and computing, the '''Levenberg–Marquardt algorithm (LMA)''',<ref name="Levenberg-Marquadt"><!--added under references heading by script-assisted edit-->
| | An individual invest loads of money things like controls actually memory cards, appear using the internet for a secondhand variation. Occasionally a store will probably feel out of used-game hardware, which could be very. Make sure you look recorded at a web-based seller's feedback serious the purchase so it is well known whether you are procuring what you covered.<br><br>The upsides of video competitions can include fun, cinema and even education. The downsides range as a result of addictive game play which will younger individuals seeing and hearing things they are not old enough over. With luck, the ideas presented within doing this article can help your manage video games clearly within your home with everyone's benefit.<br><br>Video gaming are very well-liked many homes. The most of people perform online adventures to pass through time, however, some blessed people are paid to experience clash of clans sur pc. Online gaming is going to be preferred for some evening into the future. These tips will an individual if you are looking to try out online.<br><br>Very much now, there exists little social options / attributes with this game my husband and i.e. there is not any chat, having financial problems to team track of friends, etc but manage we could expect all of this to improve soon as Boom Beach continues to stay in their Beta Mode.<br><br>Today we can use this entire operation to acquisition the [http://www.tumblr.com/tagged/majority majority] of any time together with 1hr and one year. For archetype to investment the majority of delivery up 4 a far time, acting x = 15, 400 abnormal and / or you receive y = 51 gems.<br><br>If you loved this informative article and you would like to receive much more information relating to [http://prometeu.net clash of clans hack cydia source] i implore you to visit our web page. It's difficult to select the most advisable xbox game gaming multilevel. In the beginning, you should involving your standard requirements like a video game player, accompanied by check out the specs made available from for each unit you are if you think about. Consider [http://Www.Wonderhowto.com/search/investigating+on-line/ investigating on-line]. Check many testimonials to ascertain if other useful gamers have discovered issues with the unit. To be able to buying a game process, you should know to the extent that you are able as a way to regarding it.<br><br>Evidently individuals who produced these Crack Clash of Family members are true fans related with the sport themselves, not to mention this is exactly the things ensures the potency created by our alternative, because experts needed to do the idea ourselves. |
| The algorithm was first published by Kenneth Levenberg, while working at the [[Frankford Arsenal|Frankford Army Arsenal]]. It was rediscovered by [[Donald Marquardt]] who worked as a [[statistician]] at [[DuPont]] and independently by Girard, Wynn and Morrison.</ref> also known as the '''damped least-squares (DLS)''' method, is used to solve [[non-linear least squares]] problems. These minimization problems arise especially in [[least squares]] [[curve fitting]].
| |
| | |
| The LMA interpolates between the [[Gauss–Newton algorithm]] (GNA) and the method of [[gradient descent]]. The LMA is more [[Robustness (computer science)|robust]] than the GNA, which means that in many cases it finds a solution even if it starts very far off the final minimum. For well-behaved functions and reasonable starting parameters, the LMA tends to be a bit slower than the GNA. LMA can also be viewed as Gauss–Newton using a [[trust region]] approach.
| |
| | |
| The LMA is a very popular curve-fitting algorithm used in many software applications for solving generic curve-fitting problems. However, as for many fitting algorithms, the LMA finds only a [[local minimum]], which is not necessarily the [[global minimum]].
| |
| | |
| == The problem ==
| |
| The primary application of the Levenberg–Marquardt algorithm is in the least squares curve fitting problem: given a set of m empirical datum pairs of independent and dependent variables, (''x<sub>i</sub>'', ''y<sub>i</sub>''), optimize the parameters ''β'' of the model curve ''f''(''x'','''''β''''') so that the sum of the squares of the deviations
| |
| | |
| :<math>S(\boldsymbol \beta) = \sum_{i=1}^m [y_i - f(x_i, \ \boldsymbol \beta) ]^2</math>
| |
| | |
| becomes minimal.
| |
| | |
| == The solution ==
| |
| Like other numeric minimization algorithms, the Levenberg–Marquardt algorithm is an [[iteration|iterative]] procedure. To start a minimization, the user has to provide an initial guess for the parameter vector, '''''β'''''. In cases with only one minimum, an uninformed standard guess like '''''β'''''<sup>T</sup>=(1,1,...,1) will work fine; in cases with [[local minimum|multiple minima]], the algorithm converges only if the initial guess is already somewhat close to the final solution.
| |
| | |
| In each iteration step, the parameter vector, '''''β''''', is replaced by a new estimate, '''''β''''' + '''''δ'''''. To determine '''''δ''''', the functions <math> f(x_i,\boldsymbol \beta+\boldsymbol \delta)</math> are approximated by their linearizations
| |
| | |
| : <math>f(x_i,\boldsymbol \beta+\boldsymbol \delta) \approx f(x _i,\boldsymbol \beta) + J_i \boldsymbol \delta \!</math>
| |
| | |
| where
| |
| : <math>J_i=\frac{\partial f(x_i,\boldsymbol\beta)}{\partial \boldsymbol\beta}</math>
| |
| is the [[gradient]] (row-vector in this case)
| |
| of ''f'' with respect to '''''β'''''.
| |
| | |
| At the minimum of the sum of squares, <math>S(\beta)</math>, the [[gradient]] of <math>S</math> with respect to '''''δ''''' will be zero. The above first-order approximation of <math> f(x_i,\boldsymbol \beta+\boldsymbol \delta)</math> gives
| |
| : <math>S(\boldsymbol\beta+\boldsymbol\delta) \approx \sum_{i=1}^m \left( y_i - f(x_i,\boldsymbol\beta) - J_i \boldsymbol\delta\right)^2</math>.
| |
| Or in vector notation,
| |
| : <math> S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2</math>.
| |
| Taking the derivative with respect to '''δ''' and setting the result to zero gives:
| |
| | |
| :<math>\mathbf{(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)]} \!</math>
| |
| | |
| where <math>\mathbf{J}</math> is the [[Jacobian matrix and determinant|Jacobian matrix]] whose ''i''<sup>th</sup> row equals <math>J_i</math>, and where <math>\mathbf{f}</math> and <math>\mathbf{y}</math> are vectors with ''i''<sup>th</sup> component
| |
| <math>f(x_i,\boldsymbol \beta)</math> and <math>y_i</math>, respectively.
| |
| This is a set of linear equations which can be solved for '''''δ'''''.
| |
| | |
| Levenberg's contribution is to replace this equation by a "damped version",
| |
| | |
| :<math>\mathbf{(J^{T}J + \lambda I)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)]}\!</math>
| |
| | |
| where '''I''' is the identity matrix, giving as the increment, '''''δ''''', to the estimated parameter vector, '''''β'''''.
| |
| | |
| The (non-negative) damping factor, λ, is adjusted at each iteration. If reduction of ''S'' is rapid, a smaller value can be used, bringing the algorithm closer to the [[Gauss–Newton algorithm]], whereas if an iteration gives insufficient reduction in the residual, λ can be increased, giving a step closer to the gradient descent direction. Note that the [[gradient]] of ''S'' with respect to '''''β''''' | |
| equals <math>-2(\mathbf{J}^{T} [\mathbf{y} - \mathbf{f}(\boldsymbol \beta) ] )^T</math>. Therefore, for large values of ''λ'', the step will
| |
| be taken approximately in the direction of the gradient. If either the length of the calculated step, '''''δ''''', or the reduction of sum of squares from the latest parameter vector, '''''β''''' + '''''δ''''', fall below predefined limits, iteration stops and the last parameter vector, '''''β''''', is considered to be the solution.
| |
| | |
| Levenberg's algorithm has the disadvantage that if the value of damping factor, λ, is large, inverting '''J'''<sup>T</sup>'''J''' + λ'''I''' is not used at all. Marquardt provided the insight that we can scale each component of the gradient according to the curvature so that there is larger movement along the directions where the gradient is smaller. This avoids slow convergence in the direction of small gradient. Therefore, Marquardt replaced the identity matrix, '''I''', with the diagonal matrix consisting of the diagonal elements of '''J'''<sup>T</sup>'''J''', resulting in the Levenberg–Marquardt algorithm:
| |
| | |
| :<math>\mathbf{(J^T J + \lambda\, diag(J^T J))\boldsymbol \delta = J^T [y - f(\boldsymbol \beta)]}\!</math>.
| |
| | |
| A similar damping factor appears in [[Tikhonov regularization]], which is used to solve linear [[ill-posed problems]], as well as in [[ridge regression]], an [[estimation theory|estimation]] technique in [[statistics]].
| |
| | |
| === Choice of damping parameter ===
| |
| Various more-or-less heuristic arguments have been put forward for the best choice for the damping parameter λ. Theoretical arguments exist showing why some of these choices guaranteed local convergence of the algorithm; however these choices can make the global convergence of the algorithm suffer from the undesirable properties of [[gradient descent|steepest-descent]], in particular very slow convergence close to the optimum.
| |
| | |
| The absolute values of any choice depends on how well-scaled the initial problem is. Marquardt recommended starting with a value λ<sub>0</sub> and a factor ν > 1. Initially setting λ = λ<sub>0</sub> and computing the residual sum of squares ''S''('''''β''''') after one step from the starting point with the damping factor of
| |
| λ = λ<sub>0</sub> and secondly with λ<sub>0</sub>/ν. If both of these are worse than the initial point then the damping is increased by successive multiplication by ν until a better point is found with a new damping factor of λ<sub>0</sub>ν<sup>''k''</sup> for some ''k''.
| |
| | |
| If use of the damping factor λ/ν results in a reduction in squared residual then this is taken as the new value of λ (and the new optimum location is taken as that obtained with this damping factor) and the process continues; if using λ/ν resulted in a worse residual, but using λ resulted in a better residual, then λ is left unchanged and the new optimum is taken as the value obtained with λ as damping factor.
| |
| | |
| ==Example==
| |
| | |
| [[Image:Lev-Mar-poor-fit.png|thumb|Poor fit]]
| |
| [[Image:Lev-Mar-better-fit.png|thumb|Better fit]]
| |
| [[Image:Lev-Mar-best-fit.png|thumb|Best fit]]
| |
| | |
| In this example we try to fit the function <math>y=a \cos(bX) + b \sin(aX)</math> using the Levenberg–Marquardt algorithm implemented in
| |
| [[GNU Octave]] as the ''leasqr'' function. The 3 graphs Fig 1,2,3 show progressively better fitting for the parameters ''a''=100, ''b''=102 used
| |
| in the initial curve. Only when the parameters in Fig 3 are chosen closest to the original, are the curves fitting exactly. This equation
| |
| is an example of very sensitive initial conditions for the Levenberg–Marquardt algorithm. One reason for this sensitivity is the existence of multiple minima — the function <math>\cos(\beta x)</math> has minima at parameter value <math>\hat \beta</math> and <math>\hat \beta +2n \pi.</math>
| |
| | |
| ==Other applications==
| |
| The Levenberg–Marquardt algorithm has been applied to nonlinear inverse problems. A particular application is generating computational models of oil reservoirs given the observed data.<ref>Gharib Shirangi, M., [History matching production data and uncertainty assessment with an efficient TSVD parameterization algorithm, Journal of Petroleum Science and Engineering, http://www.sciencedirect.com/science/article/pii/S0920410513003227]</ref>
| |
| | |
| ==Notes==
| |
| {{Reflist}}
| |
| | |
| == See also ==
| |
| * [[Trust region]]
| |
| * [[Nelder–Mead method]]
| |
| | |
| ==References==
| |
| <div class='references-normal'>
| |
| * {{Cite journal
| |
| | author = [[Kenneth Levenberg]]
| |
| | title = A Method for the Solution of Certain Non-Linear Problems in Least Squares
| |
| | journal = Quarterly of Applied Mathematics
| |
| | volume = 2
| |
| | pages = 164–168
| |
| | year = 1944
| |
| }}
| |
| * {{Cite journal
| |
| | author = A. Girard
| |
| | title =
| |
| | journal = Rev. Opt
| |
| | volume = 37
| |
| | pages = 225, 397
| |
| | year = 1958
| |
| }}
| |
| * {{Cite journal
| |
| | author = C.G. Wynne
| |
| | title = Lens Designing by Electronic Digital Computer: I
| |
| | journal = Proc. Phys. Soc. London
| |
| | volume = 73
| |
| | issue = 5
| |
| | pages = 777
| |
| | year = 1959
| |
| | doi = 10.1088/0370-1328/73/5/310
| |
| }}
| |
| * {{Cite journal
| |
| | author = Jorge J. Moré and Daniel C. Sorensen
| |
| | title = Computing a Trust-Region Step
| |
| | journal = SIAM J. Sci. Stat. Comput.
| |
| | pages = 553–572
| |
| | year = 1983
| |
| | issue = 4
| |
| }}
| |
| * {{Cite journal
| |
| | author = D.D. Morrison
| |
| | title =
| |
| | journal = Jet Propulsion Laboratory Seminar proceedings
| |
| | pages =
| |
| | year = 1960
| |
| }}
| |
| * {{Cite journal
| |
| | author = [[Donald Marquardt]]
| |
| | title = An Algorithm for Least-Squares Estimation of Nonlinear Parameters
| |
| | journal = SIAM Journal on Applied Mathematics
| |
| | volume = 11
| |
| | issue = 2
| |
| | pages = 431–441
| |
| | year = 1963
| |
| | doi = 10.1137/0111030
| |
| }}
| |
| * {{Cite journal
| |
| | author = Philip E. Gill and [[Walter Murray]]
| |
| | title = Algorithms for the solution of the nonlinear least-squares problem
| |
| | journal = [[SIAM Journal on Numerical Analysis]]
| |
| | volume = 15
| |
| | issue = 5
| |
| | pages = 977–992
| |
| | year = 1978
| |
| | doi = 10.1137/0715063
| |
| }}
| |
| * {{Cite journal
| |
| | author = Jose Pujol
| |
| | title = The solution of nonlinear inverse problems and the Levenberg-Marquardt method
| |
| | publisher = SEG
| |
| | year = 2007
| |
| | journal = Geophysics
| |
| | volume = 72
| |
| | number = 4
| |
| | pages = W1–W16
| |
| | url = http://link.aip.org/link/?GPY/72/W1/1
| |
| | doi = 10.1190/1.2732552
| |
| }}
| |
| * {{cite book | last = Nocedal | first = Jorge | coauthors = Wright, Stephen J. | title = Numerical Optimization, 2nd Edition | year = 2006 | publisher = Springer | isbn = 0-387-30303-0 }}
| |
| <references/> | |
| </div> | |
| | |
| == External links ==
| |
| | |
| ===Descriptions===
| |
| *Detailed description of the algorithm can be found in [http://www.nrbook.com/a/bookcpdf.php Numerical Recipes in C, Chapter 15.5: Nonlinear models]
| |
| *C. T. Kelley, ''Iterative Methods for Optimization'', SIAM Frontiers in Applied Mathematics, no 18, 1999, ISBN 0-89871-433-8. [http://www.siam.org/books/textbooks/fr18_book.pdf Online copy]
| |
| *[http://www3.villanova.edu/maple/misc/mtc1093.html History of the algorithm in SIAM news]
| |
| *[http://ananth.in/docs/lmtut.pdf A tutorial by Ananth Ranganathan]
| |
| *[http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3215/pdf/imm3215.pdf Methods for Non-Linear Least Squares Problems] by K. Madsen, H.B. Nielsen, O. Tingleff is a tutorial discussing non-linear least-squares in general and the Levenberg-Marquardt method in particular
| |
| * T. Strutz: ''Data Fitting and Uncertainty (A practical introduction to weighted least squares and beyond)''. Vieweg+Teubner, ISBN 978-3-8348-1022-9.
| |
| | |
| === Implementations ===
| |
| * Levenberg-Marquardt is a built-in algorithm in [[Mathematica]] <!-- <ref>[http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationIntroductionLocalMinimization.html Unconstrained optimization methods in Mathematica.</ref> -->, [[Matlab]], [[GNU Octave]], [[Origin (software)|Origin]], and [[IGOR Pro]].
| |
| *The oldest implementation still in use is [http://www.netlib.org/minpack/ lmdif], from [[MINPACK]], in [[Fortran]], in the [[public domain]]. See also:
| |
| **[http://apps.jcns.fz-juelich.de/lmfit lmfit], a self-contained [[C programming language|C]] implementation of the MINPACK algorithm, with an easy-to-use wrapper for curve fitting, liberal licence (freeBSD).
| |
| ** [http://eigen.tuxfamily.org/index.php?title=Main_Page eigen], a C++ linear algebra library, includes an adaptation of the minpack algorithm in the "NonLinearOptimization" module.
| |
| **The [[GNU Scientific Library]] library has a C interface to MINPACK.
| |
| **[http://devernay.free.fr/hacks/cminpack.html C/C++ Minpack] includes the Levenberg–Marquardt algorithm.
| |
| **Several high-level languages and mathematical packages have wrappers for the [[MINPACK]] routines, among them:
| |
| ***Python library [[scipy]], module <code>scipy.optimize.leastsq</code>,
| |
| ***[[IDL (programming language)|IDL]], add-on [http://cow.physics.wisc.edu/~craigm/idl/fitting.html MPFIT].
| |
| ***[[R (programming language)]] has the [http://cran.r-project.org/web/packages/minpack.lm/index.html minpack.lm] package.
| |
| *[http://www.ics.forth.gr/%7elourakis/levmar/ levmar] is an implementation in [[C (programming language)|C]]/[[C++]] with support for constraints, distributed under the [[GNU General Public License]].
| |
| **levmar includes a [[MEX file]] interface for [[MATLAB]]
| |
| **[[Perl]] ([[Perl Data Language|PDL]]), [[Python (programming language)|python]], [[Haskell (programming language)|Haskell]] and [[.NET Framework|.NET]] interfaces to levmar are available: see [http://www.johnlapeyre.com/pdl/index.html PDL::Fit::Levmar] or [https://metacpan.org/module/PDL::Fit::LM PDL::Fit::LM], [http://trac.astrometry.net/wiki/PyLevmar PyLevmar], [http://hackage.haskell.org/package/levmar HackageDB levmar] and [https://github.com/AvengerDr/LevmarSharp LevmarSharp].
| |
| *[http://www.ics.forth.gr/%7elourakis/sparseLM/ sparseLM] is a [[C (programming language)|C]] implementation aimed at minimizing functions with large, arbitrarily [[Sparse matrix|sparse]] Jacobians. Includes a MATLAB MEX interface.
| |
| *[http://www2.imm.dtu.dk/~hbni/Software/SMarquardt.m SMarquardt.m] is a stand-alone routine for Matlab or Octave.
| |
| *[http://www.bnikolic.co.uk/inmin/inmin-library.html InMin] library contains a C++ implementation of the algorithm based on the [http://eigen.tuxfamily.org/index.php?title=Main_Page eigen] C++ linear algebra library. It has a pure C-language API as well as a Python binding
| |
| *[http://code.google.com/p/ceres-solver/ ceres] is a non-linear minimisation library with an implementation of the Levenberg–Marquardt algorithm. It is written in C++ and uses [http://eigen.tuxfamily.org/index.php?title=Main_Page eigen]
| |
| *[http://www.alglib.net/optimization/levenbergmarquardt.php ALGLIB] has implementations of improved LMA in C# / C++ / Delphi / Visual Basic. Improved algorithm takes less time to converge and can use either Jacobian or exact Hessian.
| |
| *[[NMath]] has an implementation for the [[.NET Framework]].
| |
| *[[gnuplot]] uses its own implementation [http://www.gnuplot.info/ gnuplot.info].
| |
| *[[Java (programming language)|Java programming language]] implementations: 1) [http://scribblethink.org/Computer/Javanumeric/index.html Javanumerics], 2) [http://virtualrisk.cvs.sourceforge.net/*checkout*/virtualrisk/util/lma/lma_v1.3.zip LMA-package] (a small, user friendly and well documented implementation with examples and support), 3) [http://commons.apache.org/math/apidocs/org/apache/commons/math/optimization/general/LevenbergMarquardtOptimizer.html Apache Commons Math]
| |
| *[http://oooconv.free.fr/fitoo/fitoo_en.html OOoConv] implements the L-M algorithm as an OpenOffice.org Calc spreadsheet.
| |
| *[[SAS (software)|SAS]], there are multiple ways to access SAS's implementation of the Levenberg–Marquardt algorithm: it can be accessed via [http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/langref_sect187.htm#imlug_langref_nlplm NLPLM Call] in [http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/imlstart_sect1.htm PROC IML] and it can also be accessed through the [http://support.sas.com/documentation/cdl/en/ormpug/63352/HTML/default/viewer.htm#ormpug_nlp_sect021.htm LSQ] statement in [http://support.sas.com/documentation/cdl/en/ormpug/63352/HTML/default/viewer.htm#ormpug_nlp_sect001.htm PROC NLP], and the [http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_nlin_sect007.htm METHOD=MARQUARDT] option in [http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_nlin_sect001.htm PROC NLIN].
| |
| | |
| {{Optimization algorithms}}
| |
| | |
| {{DEFAULTSORT:Levenberg-Marquardt algorithm}}
| |
| [[Category:Statistical algorithms]]
| |
| [[Category:Optimization algorithms and methods]]
| |
| [[Category:Least squares]]
| |