Line moiré

In statistics, Fisher's scoring algorithm is a form of Newton's method used to solve maximum likelihood equations numerically.

Sketch of Derivation

Let $Y_1,\dots ,Y_n$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) ${\theta }^{*}$ of $\theta$. First, suppose we have a starting point for our algorithm ${\theta }_0$, and consider a Taylor expansion of the score function, $V(\theta )$, about ${\theta }_0$:

$V(\theta )\approx V({\theta }_0)-\mathcal{𝒥}({\theta }_0)(\theta -{\theta }_0),$

where

$\mathcal{𝒥}({\theta }_0)=-\sum _{i=1}^n{\mathrm{\nabla }{\mathrm{\nabla }}^{\mathrm{\top }}|}_{\theta ={\theta }_0}\log f(Y_i;\theta )$

is the observed information matrix at ${\theta }_0$. Now, setting $\theta ={\theta }^{*}$, using that $V({\theta }^{*})=0$ and rearranging gives us:

${\theta }^{*}\approx {\theta }_0+{\mathcal{𝒥}}^{-1}({\theta }_0)V({\theta }_0).$

We therefore use the algorithm

${\theta }_{m+1}={\theta }_m+{\mathcal{𝒥}}^{-1}({\theta }_m)V({\theta }_m),$

and under certain regularity conditions, it can be shown that ${\theta }_m\to {\theta }^{*}$.

Fisher scoring

In practice, $\mathcal{𝒥}(\theta )$ is usually replaced by $\mathcal{ℐ}(\theta )=\mathrm{E}[\mathcal{𝒥}(\theta )]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

${\theta }_{m+1}={\theta }_m+{\mathcal{ℐ}}^{-1}({\theta }_m)V({\theta }_m)$.

See also

• Score (statistics)

References

Jennrich, R. I., & Sampson, P. F. (1976). Newton-Raphson and related algorithms for maximum likelihood variance component estimation. Technometrics, 18, 11-17.