Tomlinson model

In probability and statistics, the Hellinger distance is used to quantify the similarity between two probability distributions. It is a type of f-divergence. The Hellinger distance is defined in terms of the Hellinger integral, which was introduced by Ernst Hellinger in 1909.

Definition

Measure theory

To define the Hellinger distance in terms of measure theory, let P and Q denote two probability measures that are absolutely continuous with respect to a third probability measure λ. The square of the Hellinger distance between P and Q is defined as the quantity

$H^2(P,Q)=\frac{1}{2}\int {(\sqrt{\frac{dP}{d\lambda }}-\sqrt{\frac{dQ}{d\lambda }})}^{2}d\lambda .$

Here, dP / dλ and dQ / dλ are the Radon–Nikodym derivatives of P and Q respectively. This definition does not depend on λ, so the Hellinger distance between P and Q does not change if λ is replaced with a different probability measure with respect to which both P and Q are absolutely continuous. For compactness, the above formula is often written as

$H^2(P,Q)=\frac{1}{2}\int {(\sqrt{dP}-\sqrt{dQ})}^2.$

Probability theory using Lebesgue measure

To define the Hellinger distance in terms of elementary probability theory, we take λ to be Lebesgue measure, so that dP / dλ and dQ / dλ are simply probability density functions. If we denote the densities as f and g, respectively, the squared Hellinger distance can be expressed as a standard calculus integral

$\frac{1}{2}\int {(\sqrt{f(x)}-\sqrt{g(x)})}^{2}dx=1-\int \sqrt{f(x)g(x)}dx,$

where the second form can be obtained by expanding the square and using the fact that the integral of a probability density over its domain must be one.

The Hellinger distance H(P, Q) satisfies the property (derivable from the Cauchy-Schwarz inequality)

$0\le H(P,Q)\le 1.$

Discrete distributions

For two discrete probability distributions $P=(p_1\dots p_k)$ and $Q=(q_1\dots q_k)$, their Hellinger distance is defined as

$H(P,Q)=\frac{1}{\sqrt{2}}\sqrt{\sum _{i=1}^k(\sqrt{p_i}-\sqrt{q_i}{)}^2},$

which is directly related to the Euclidean norm of the difference of the square root vectors, i.e.

$H(P,Q)=\frac{1}{\sqrt{2}}\Vert \sqrt{P}-\sqrt{Q}{\Vert }_2.$

Connection with the statistical distance

The Hellinger distance $H(P,Q)$ and the total variation distance (or statistical distance) $\delta (P,Q)$ are related as follows:

$H^2(P,Q)\le \delta (P,Q)\le \sqrt{2}H(P,Q).$

These inequalities follow immediately from the inequalities between the 1-norm and the 2-norm.

Properties

The maximum distance 1 is achieved when P assigns probability zero to every set to which Q assigns a positive probability, and vice versa.

Sometimes the factor 1/2 in front of the integral is omitted, in which case the Hellinger distance ranges from zero to the square root of two.

The Hellinger distance is related to the Bhattacharyya coefficient $BC(P,Q)$ as it can be defined as

$H(P,Q)=\sqrt{1-BC(P,Q)}.$

Hellinger distances are used in the theory of sequential and asymptotic statistics.

Examples

The squared Hellinger distance between two normal distributions $P\sim \mathcal{𝒩}({\mu }_1,{\sigma }_1^2)$ and $Q\sim \mathcal{𝒩}({\mu }_2,{\sigma }_2^2)$ is:

$H^2(P,Q)=1-\sqrt{\frac{2{\sigma }_1{\sigma }_2}{{\sigma }_1^2+{\sigma }_2^2}}{e}^{-\frac{1}{4}\frac{({\mu }_1-{\mu }_2{)}^2}{{\sigma }_1^2+{\sigma }_2^2}}.$

The squared Hellinger distance between two exponential distributions $P\sim \mathrm{E}\mathrm{x}\mathrm{p}\mathrm{(}\mathrm{\alpha }\mathrm{)}$ and $Q\sim \mathrm{E}\mathrm{x}\mathrm{p}\mathrm{(}\mathrm{\beta }\mathrm{)}$ is:

$H^2(P,Q)=1-\frac{2\sqrt{\alpha \beta }}{\alpha +\beta }.$

The squared Hellinger distance between two Weibull distributions $P\sim \mathrm{W}\mathrm{(}\mathrm{k},\mathrm{\alpha }\mathrm{)}$ and $Q\sim \mathrm{W}\mathrm{(}\mathrm{k},\mathrm{\beta }\mathrm{)}$ (where $k$ is a common shape parameter and $\alpha ,\beta$ are the scale parameters respectively):

$H^2(P,Q)=1-\frac{2(\alpha \beta {)}^{k/2}}{{\alpha }^k+{\beta }^k}.$

The squared Hellinger distance between two Poisson distributions with rate parameters $\alpha$ and $\beta$, so that $P\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\mathrm{(}\mathrm{\alpha }\mathrm{)}$ and $Q\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\mathrm{(}\mathrm{\beta }\mathrm{)}$, is:

$H^2(P,Q)=1-e^{-\frac{1}{2}(\sqrt{\alpha }-\sqrt{\beta }{)}^2}.$

See also

• Fisher information metric

Notes

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

References

• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534
• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534
• 20 year-old Real Estate Agent Rusty from Saint-Paul, has hobbies and interests which includes monopoly, property developers in singapore and poker. Will soon undertake a contiki trip that may include going to the Lower Valley of the Omo.
  
  My blog: http://www.primaboinca.com/view_profile.php?userid=5889534