Scoring algorithm

Scoring algorithm, also known as Fisher's scoring, is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.

Sketch of derivation

Let $Y_1,\dots ,Y_n$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) ${\theta }^{*}$ of $\theta$. First, suppose we have a starting point for our algorithm ${\theta }_0$, and consider a Taylor expansion of the score function, $V(\theta )$, about ${\theta }_0$:

$V(\theta )\approx V({\theta }_0)-𝒥({\theta }_0)(\theta -{\theta }_0),$

where

$𝒥({\theta }_0)=-\sum _{i=1}^n{\mathrm{\nabla }{\mathrm{\nabla }}^{\mathrm{\top }}|}_{\theta ={\theta }_0}\log f(Y_i;\theta )$

is the observed information matrix at ${\theta }_0$. Now, setting $\theta ={\theta }^{*}$, using that $V({\theta }^{*})=0$ and rearranging gives us:

${\theta }^{*}\approx {\theta }_0+{𝒥}^{-1}({\theta }_0)V({\theta }_0).$

We therefore use the algorithm

${\theta }_{m+1}={\theta }_m+{𝒥}^{-1}({\theta }_m)V({\theta }_m),$

and under certain regularity conditions, it can be shown that ${\theta }_m\to {\theta }^{*}$.

Fisher scoring

In practice, $𝒥(\theta )$ is usually replaced by $ℐ(\theta )=\mathrm{E}[𝒥(\theta )]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

${\theta }_{m+1}={\theta }_m+{ℐ}^{-1}({\theta }_m)V({\theta }_m)$..

Under some regularity conditions, if ${\theta }_m$ is a consistent estimator, then ${\theta }_{m+1}$ (the correction after a single step) is 'optimal' in the sense that its error distribution is asymptotically identical to that of the true max-likelihood estimate.

See also

• Score (statistics)
• Score test
• Fisher information