Back-stripping

In coding theory, Kraft's inequality, named after Leon Kraft, gives a sufficient condition for the existence of a prefix code and necessary condition for the existence of a uniquely decodable code for a given set of codeword lengths. Its applications to prefix codes and trees often find use in computer science and information theory.

More specifically, Kraft's inequality limits the lengths of codewords in a prefix code: if one takes an exponential of the length of each valid codeword, the resulting set of values must look like a probability mass function, that is, it must have total measure less than or equal to one. Kraft's inequality can be thought of in terms of a constrained budget to be spent on codewords, with shorter codewords being more expensive.

• If Kraft's inequality holds with strict inequality, the code has some redundancy.
• If Kraft's inequality holds with equality, the code in question is a complete code.
• If Kraft's inequality does not hold, the code is not uniquely decodable.

Kraft's inequality was published by Template:Harvtxt. However, Kraft's paper discusses only prefix codes, and attributes the analysis leading to the inequality to Raymond Redheffer. The inequality is sometimes also called the Kraft–McMillan theorem after the independent discovery of the result by Template:Harvtxt; McMillan proves the result for the general case of uniquely decodable codes, and attributes the version for prefix codes to a spoken observation in 1955 by Joseph Leo Doob.

Examples

Binary trees

File:AVLtreef.svg

9, 14, 19, 67 and 76 are leaf nodes at depths of 3, 3, 3, 3 and 2, respectively.

Any binary tree can be viewed as defining a prefix code for the leaves of the tree. Kraft's inequality states that

$\sum _{\ell \in \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{s}}{2}^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}\le 1.$

Here the sum is taken over the leaves of the tree, i.e. the nodes without any children. The depth is the distance to the root node. In the tree to the right, this sum is

$\frac{1}{4}+4\left(\frac{1}{8}\right)=\frac{3}{4}\le 1.$

Chaitin's constant

In algorithmic information theory, Chaitin's constant is defined as

$\mathrm{\Omega }=\sum _{p\in P}{2}^{-\left|p\right|}.$

This is an infinite sum, which has one summand for every syntactically correct program that halts. |p| stands for the length of the bit string of p. The programs are required to be prefix-free in the sense that no summand has a prefix representing a syntactically valid program that halts. Hence the bit strings are prefix codes, and Kraft's inequality gives that $\mathrm{\Omega }\le 1$.

Formal statement

Let each source symbol from the alphabet

$S=\{s_1,s_2,\dots ,s_n\}$

be encoded into a uniquely decodable code over an alphabet of size $r$ with codeword lengths

${\ell }_1,{\ell }_2,\dots ,{\ell }_n.$

Then

$\sum _{i=1}^n{\left(\frac{1}{r}\right)}^{{\ell }_i}\le 1.$

Conversely, for a given set of natural numbers ${\ell }_1,{\ell }_2,\dots ,{\ell }_n$ satisfying the above inequality, there exists a uniquely decodable code over an alphabet of size $r$ with those codeword lengths.

A commonly occurring special case of a uniquely decodable code is a prefix code. Kraft's inequality therefore also holds for any prefix code.

Proof for prefix codes

File:Kraft inequality example.png

Example for binary tree. Red nodes represent a prefix tree. The method for calculating the number of descendant leaf nodes in the full tree is shown.

Suppose that ${\ell }_1\le {\ell }_2\le ...\le {\ell }_n$. Let $A$ be the full $r$-ary tree of depth ${\ell }_n$. Every word of length $\ell \le {\ell }_n$ over an $r$-ary alphabet corresponds to a node in this tree at depth $\ell$. The $i$th word in the prefix code corresponds to a node $v_i$; let $A_i$ be the set of all leaf nodes in the subtree of $A$ rooted at $v_i$. Clearly

$\left|A_i\right|=r^{{\ell }_n-{\ell }_i}.$

Since the code is a prefix code,

$A_i\cap A_j=\varnothing ,i\ne j$.

Thus, given that the total number of nodes at depth ${\ell }_n$ is $r^{{\ell }_n}$,

$\left|\bigcup _{i=1}^n{A}_i\right|=\sum _{i=1}^n{r}^{{\ell }_n-{\ell }_i}\le r^{{\ell }_n}$

from which the result follows.

Conversely, given any ordered sequence of $n$ natural numbers,

${\ell }_1\le {\ell }_2\le \dots \le {\ell }_n$

satisfying the Kraft inequality, one can construct a prefix code with codeword lengths equal to ${\ell }_i$ by pruning subtrees from a full $r$-ary tree of depth ${\ell }_n$. First choose any node from the full tree at depth ${\ell }_1$ and remove all of its descendants. This removes $r^{-{\ell }_1}$ fraction of the nodes from the full tree from being considered for the rest of the remaining codewords. The next iteration removes $r^{-{\ell }_2}$ fraction of the full tree for total of $r^{-{\ell }_1}+r^{-{\ell }_2}$. After $m$ iterations,

$\sum _{i=1}^m{r}^{-{\ell }_i}$

fraction of the full tree nodes are removed from consideration for any remaining codewords. But, by the assumption, this sum is less than 1 for all $m<n$. Thus prefix code with lengths ${\ell }_i$ can be constructed for all $n$ source symbols.

Proof for binary trees

Let $T$ be any binary tree and let $T^{\prime }$ be the binary tree obtained by attaching a leaf to any node of $T$ that has exactly one child so that $T^{\prime }$ is full; every node of $T^{\prime }$ has either two or zero children. Every leaf of $T$ is present in $T^{\prime }$ so we immediately have

$\sum _{\ell \in \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{s}(T)}{2}^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}\le \sum _{\ell \in \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{s}(T^{\prime })}{2}^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}.$

Consider the random walk that starts at the root of $T^{\prime }$ and repeatedly moves to the left or right child of the current node, with equal probability, until it reaches a leaf. The probability that this walk reaches a particular leaf, $\ell$, is exactly $2^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}$. Therefore, $⟨2^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}:\ell \in \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{s}(T^{\prime })⟩$ is a probability distribution so

$\sum _{\ell \in \mathrm{l}\mathrm{e}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{s}(T^{\prime })}{2}^{-\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(\ell )}=1.$

The proof of the converse half of the result is given above.

Proof of the general case

Consider the generating function in inverse of x for the code S

$F(x)=\sum _{i=1}^n{x}^{-\left|s_i\right|}=\sum _{\ell =\min }^{\max }{p}_{\ell }{x}^{-\ell }$

in which $p_{\ell }$—the coefficient in front of $x^{-\ell }$—is the number of distinct codewords of length $\ell$. Here min is the length of the shortest codeword in S, and max is the length of the longest codeword in S.

For any positive integer m consider the m-fold product S^m, which consists of all the words of the form $s_{i_1}{s}_{i_2}\dots s_{i_m}$, where $i_1,i_2,\dots ,i_m$ are indices between 1 and n. Note that, since S was assumed to uniquely decodable, if $s_{i_1}{s}_{i_2}\dots s_{i_m}=s_{j_1}{s}_{j_2}\dots s_{j_m}$, then $i_1=j_1,i_2=j_2,\dots ,i_m=j_m$. In other words, every word in $S^m$ comes from a unique sequence of codewords in $S$. Because of this property, one can compute the generating function $G(x)$ for $S^m$ from the generating function $F(x)$ as

$G(x)={(F(x))}^m={\left(\sum _{i=1}^n{x}^{-\left|s_i\right|}\right)}^m=\sum _{i_1=1}^n\sum _{i_2=1}^n\cdots \sum _{i_m=1}^n{x}^{-\left(\right|s_{i_1}|+|s_{i_2}|+\cdots +|s_{i_m}\left|\right)}=\sum _{i_1=1}^n\sum _{i_2=1}^n\cdots \sum _{i_m=1}^n{x}^{-|s_{i_1}{s}_{i_2}\cdots s_{i_m}|}=\sum _{\ell =m\cdot \min }^{m\cdot \max }{q}_{\ell }{x}^{-\ell }.$

Here, similarly as before, $q_{\ell }$—the coefficient in front of $x^{-\ell }$ in $G(x)$—is the number of words of length $\ell$ in $S^m$. Clearly, $q_{\ell }$ cannot exceed $r^{\ell }$. Hence for any positive x

${(F(x))}^m\le \sum _{\ell =m\cdot \min }^{m\cdot \max }{r}^{\ell }{x}^{-\ell }.$

Substituting the value x = r we have

${(F(r))}^m\le m\cdot (\max -\min )+1$

for any positive integer $m$. The left side of the inequality grows exponentially in $m$ and the right side only linearly. The only possibility for the inequality to be valid for all $m$ is that $F(r)\le 1$. Looking back on the definition of $F(x)$ we finally get the inequality.

$\sum _{i=1}^n{r}^{-{\ell }_i}=\sum _{i=1}^n{r}^{-\left|s_i\right|}=F(r)\le 1.$

Notes

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

References

• Many property agents need to declare for the PIC grant in Singapore. However, not all of them know find out how to do the correct process for getting this PIC scheme from the IRAS. There are a number of steps that you need to do before your software can be approved.
  
  Naturally, you will have to pay a safety deposit and that is usually one month rent for annually of the settlement. That is the place your good religion deposit will likely be taken into account and will kind part or all of your security deposit. Anticipate to have a proportionate amount deducted out of your deposit if something is discovered to be damaged if you move out. It's best to you'll want to test the inventory drawn up by the owner, which can detail all objects in the property and their condition. If you happen to fail to notice any harm not already mentioned within the inventory before transferring in, you danger having to pay for it yourself.
  
  In case you are in search of an actual estate or Singapore property agent on-line, you simply should belief your intuition. It's because you do not know which agent is nice and which agent will not be. Carry out research on several brokers by looking out the internet. As soon as if you end up positive that a selected agent is dependable and reliable, you can choose to utilize his partnerise in finding you a home in Singapore. Most of the time, a property agent is taken into account to be good if he or she locations the contact data on his website. This may mean that the agent does not mind you calling them and asking them any questions relating to new properties in singapore in Singapore. After chatting with them you too can see them in their office after taking an appointment.
  
  Have handed an trade examination i.e Widespread Examination for House Brokers (CEHA) or Actual Property Agency (REA) examination, or equal; Exclusive brokers are extra keen to share listing information thus making certain the widest doable coverage inside the real estate community via Multiple Listings and Networking. Accepting a severe provide is simpler since your agent is totally conscious of all advertising activity related with your property. This reduces your having to check with a number of agents for some other offers. Price control is easily achieved. Paint work in good restore-discuss with your Property Marketing consultant if main works are still to be done. Softening in residential property prices proceed, led by 2.8 per cent decline within the index for Remainder of Central Region
  
  Once you place down the one per cent choice price to carry down a non-public property, it's important to accept its situation as it is whenever you move in – faulty air-con, choked rest room and all. Get round this by asking your agent to incorporate a ultimate inspection clause within the possibility-to-buy letter. HDB flat patrons routinely take pleasure in this security net. "There's a ultimate inspection of the property two days before the completion of all HDB transactions. If the air-con is defective, you can request the seller to repair it," says Kelvin.
  
  15.6.1 As the agent is an intermediary, generally, as soon as the principal and third party are introduced right into a contractual relationship, the agent drops out of the image, subject to any problems with remuneration or indemnification that he could have against the principal, and extra exceptionally, against the third occasion. Generally, agents are entitled to be indemnified for all liabilities reasonably incurred within the execution of the brokers´ authority.
  
  To achieve the very best outcomes, you must be always updated on market situations, including past transaction information and reliable projections. You could review and examine comparable homes that are currently available in the market, especially these which have been sold or not bought up to now six months. You'll be able to see a pattern of such report by clicking here It's essential to defend yourself in opposition to unscrupulous patrons. They are often very skilled in using highly unethical and manipulative techniques to try and lure you into a lure. That you must also protect your self, your loved ones, and personal belongings as you'll be serving many strangers in your home. Sign a listing itemizing of all of the objects provided by the proprietor, together with their situation. HSR Prime Recruiter 2010.

• Many property agents need to declare for the PIC grant in Singapore. However, not all of them know find out how to do the correct process for getting this PIC scheme from the IRAS. There are a number of steps that you need to do before your software can be approved.
  
  Naturally, you will have to pay a safety deposit and that is usually one month rent for annually of the settlement. That is the place your good religion deposit will likely be taken into account and will kind part or all of your security deposit. Anticipate to have a proportionate amount deducted out of your deposit if something is discovered to be damaged if you move out. It's best to you'll want to test the inventory drawn up by the owner, which can detail all objects in the property and their condition. If you happen to fail to notice any harm not already mentioned within the inventory before transferring in, you danger having to pay for it yourself.
  
  In case you are in search of an actual estate or Singapore property agent on-line, you simply should belief your intuition. It's because you do not know which agent is nice and which agent will not be. Carry out research on several brokers by looking out the internet. As soon as if you end up positive that a selected agent is dependable and reliable, you can choose to utilize his partnerise in finding you a home in Singapore. Most of the time, a property agent is taken into account to be good if he or she locations the contact data on his website. This may mean that the agent does not mind you calling them and asking them any questions relating to new properties in singapore in Singapore. After chatting with them you too can see them in their office after taking an appointment.
  
  Have handed an trade examination i.e Widespread Examination for House Brokers (CEHA) or Actual Property Agency (REA) examination, or equal; Exclusive brokers are extra keen to share listing information thus making certain the widest doable coverage inside the real estate community via Multiple Listings and Networking. Accepting a severe provide is simpler since your agent is totally conscious of all advertising activity related with your property. This reduces your having to check with a number of agents for some other offers. Price control is easily achieved. Paint work in good restore-discuss with your Property Marketing consultant if main works are still to be done. Softening in residential property prices proceed, led by 2.8 per cent decline within the index for Remainder of Central Region
  
  Once you place down the one per cent choice price to carry down a non-public property, it's important to accept its situation as it is whenever you move in – faulty air-con, choked rest room and all. Get round this by asking your agent to incorporate a ultimate inspection clause within the possibility-to-buy letter. HDB flat patrons routinely take pleasure in this security net. "There's a ultimate inspection of the property two days before the completion of all HDB transactions. If the air-con is defective, you can request the seller to repair it," says Kelvin.
  
  15.6.1 As the agent is an intermediary, generally, as soon as the principal and third party are introduced right into a contractual relationship, the agent drops out of the image, subject to any problems with remuneration or indemnification that he could have against the principal, and extra exceptionally, against the third occasion. Generally, agents are entitled to be indemnified for all liabilities reasonably incurred within the execution of the brokers´ authority.
  
  To achieve the very best outcomes, you must be always updated on market situations, including past transaction information and reliable projections. You could review and examine comparable homes that are currently available in the market, especially these which have been sold or not bought up to now six months. You'll be able to see a pattern of such report by clicking here It's essential to defend yourself in opposition to unscrupulous patrons. They are often very skilled in using highly unethical and manipulative techniques to try and lure you into a lure. That you must also protect your self, your loved ones, and personal belongings as you'll be serving many strangers in your home. Sign a listing itemizing of all of the objects provided by the proprietor, together with their situation. HSR Prime Recruiter 2010.

External links

• Kraft's inequality (NIST)