# Kraft's inequality

In coding theory, Kraft's inequality, named after Leon Kraft, gives a sufficient condition for the existence of a prefix code and necessary condition for the existence of a uniquely decodable code for a given set of codeword lengths. Its applications to prefix codes and trees often find use in computer science and information theory.

More specifically, Kraft's inequality limits the lengths of codewords in a prefix code: if one takes an exponential of the length of each valid codeword, the resulting set of values must look like a probability mass function, that is, it must have total measure less than or equal to one. Kraft's inequality can be thought of in terms of a constrained budget to be spent on codewords, with shorter codewords being more expensive.

• If Kraft's inequality holds with strict inequality, the code has some redundancy.
• If Kraft's inequality holds with equality, the code in question is a complete code.
• If Kraft's inequality does not hold, the code is not uniquely decodable.

Kraft's inequality was published by Template:Harvtxt. However, Kraft's paper discusses only prefix codes, and attributes the analysis leading to the inequality to Raymond Redheffer. The inequality is sometimes also called the Kraft–McMillan theorem after the independent discovery of the result by Template:Harvtxt; McMillan proves the result for the general case of uniquely decodable codes, and attributes the version for prefix codes to a spoken observation in 1955 by Joseph Leo Doob.

## Examples

### Binary trees

Any binary tree can be viewed as defining a prefix code for the leaves of the tree. Kraft's inequality states that

$\sum _{\ell \in \mathrm {leaves} }2^{-\mathrm {depth} (\ell )}\leq 1.$ Here the sum is taken over the leaves of the tree, i.e. the nodes without any children. The depth is the distance to the root node. In the tree to the right, this sum is

${\frac {1}{4}}+4\left({\frac {1}{8}}\right)={\frac {3}{4}}\leq 1.$ ### Chaitin's constant

In algorithmic information theory, Chaitin's constant is defined as

$\Omega =\sum _{p\in P}2^{-|p|}.$ This is an infinite sum, which has one summand for every syntactically correct program that halts. |p| stands for the length of the bit string of p. The programs are required to be prefix-free in the sense that no summand has a prefix representing a syntactically valid program that halts. Hence the bit strings are prefix codes, and Kraft's inequality gives that $\Omega \leq 1$ .

## Formal statement

Let each source symbol from the alphabet

$S=\{\,s_{1},s_{2},\ldots ,s_{n}\,\}\,$ be encoded into a uniquely decodable code over an alphabet of size $r$ with codeword lengths

$\ell _{1},\ell _{2},\ldots ,\ell _{n}.\,$ Then

$\sum _{i=1}^{n}\left({\frac {1}{r}}\right)^{\ell _{i}}\leq 1.$ Conversely, for a given set of natural numbers $\ell _{1},\ell _{2},\ldots ,\ell _{n}\,$ satisfying the above inequality, there exists a uniquely decodable code over an alphabet of size $r$ with those codeword lengths.

A commonly occurring special case of a uniquely decodable code is a prefix code. Kraft's inequality therefore also holds for any prefix code.

## Proof

### Proof for prefix codes Example for binary tree. Red nodes represent a prefix tree. The method for calculating the number of descendant leaf nodes in the full tree is shown.
$|A_{i}|=r^{\ell _{n}-\ell _{i}}.$ Since the code is a prefix code,

$A_{i}\cap A_{j}=\varnothing ,\quad i\neq j$ .
$|\bigcup _{i=1}^{n}A_{i}|=\sum _{i=1}^{n}r^{\ell _{n}-\ell _{i}}\leq r^{\ell _{n}}$ from which the result follows.

Conversely, given any ordered sequence of $n$ natural numbers,

$\ell _{1}\leq \ell _{2}\leq \dots \leq \ell _{n}$ satisfying the Kraft inequality, one can construct a prefix code with codeword lengths equal to $\ell _{i}$ by pruning subtrees from a full $r$ -ary tree of depth $\ell _{n}$ . First choose any node from the full tree at depth $\ell _{1}$ and remove all of its descendants. This removes $r^{-\ell _{1}}$ fraction of the nodes from the full tree from being considered for the rest of the remaining codewords. The next iteration removes $r^{-\ell _{2}}$ fraction of the full tree for total of $r^{-\ell _{1}}+r^{-\ell _{2}}$ . After $m$ iterations,

$\sum _{i=1}^{m}r^{-\ell _{i}}$ fraction of the full tree nodes are removed from consideration for any remaining codewords. But, by the assumption, this sum is less than 1 for all $m . Thus prefix code with lengths $\ell _{i}$ can be constructed for all $n$ source symbols.

### Probabilistic Proof for prefix codes

Generate a sequence of symbols from the r character alphabet, independently and uniformly at random. Define Ei to be the event that codeword i is a prefix of this sequence. Because we have a prefix code, these events are mutually exclusive. Therefore,

The proof of the converse half of the result is given above.

### Proof of the general case

Consider the generating function in inverse of x for the code S

$F(x)=\sum _{i=1}^{n}x^{-|s_{i}|}=\sum _{\ell =\min }^{\max }p_{\ell }\,x^{-\ell }$ in which $p_{\ell }$ —the coefficient in front of $x^{-\ell }$ —is the number of distinct codewords of length $\ell$ . Here min is the length of the shortest codeword in S, and max is the length of the longest codeword in S.

$G(x)=\left(F(x)\right)^{m}=\left(\sum _{i=1}^{n}x^{-|s_{i}|}\right)^{m}=$ $=\sum _{i_{1}=1}^{n}\sum _{i_{2}=1}^{n}\cdots \sum _{i_{m}=1}^{n}x^{-\left(|s_{i_{1}}|+|s_{i_{2}}|+\cdots +|s_{i_{m}}|\right)}=$ $=\sum _{i_{1}=1}^{n}\sum _{i_{2}=1}^{n}\cdots \sum _{i_{m}=1}^{n}x^{-|s_{i_{1}}s_{i_{2}}\cdots s_{i_{m}}|}=\sum _{\ell =m\cdot \min }^{m\cdot \max }q_{\ell }\,x^{-\ell }\;.$ $\left(F(x)\right)^{m}\leq \sum _{\ell =m\cdot \min }^{m\cdot \max }r^{\ell }\,x^{-\ell }\;.$ Substituting the value x = r we have

$\left(F(r)\right)^{m}\leq m\cdot (\max -\min )+1$ for any positive integer $m$ . The left side of the inequality grows exponentially in $m$ and the right side only linearly. The only possibility for the inequality to be valid for all $m$ is that $F(r)\leq 1$ . Looking back on the definition of $F(x)$ we finally get the inequality.

$\sum _{i=1}^{n}r^{-\ell _{i}}=\sum _{i=1}^{n}r^{-|s_{i}|}=F(r)\leq 1\;.$ ### Alternative construction for the converse

$\ell _{1}\leq \ell _{2}\leq \dots \leq \ell _{n}$ satisfying the Kraft inequality, we can construct a prefix code as follows. Define the ith codeword, Ci, to be the first li digits after the radix point (e.g. decimal point) in the base r representation of

Note that by Kraft's inequality, this sum is never more than 1. Hence the codewords capture the entire value of the sum. Therefore, for j > i, the first li digits of Cj form a larger number than Ci, so the code is prefix free.