next up previous contents
Next: First and second order Up: Constraint induced correlations Previous: Distribution of the observed

 

High order entropies of the observed states

The entropy at order m is given by
 eqnarray1293
where in the second equation tex2html_wrap_inline13747 may be understood as the probability of any observed m bit state having b one bits - all such states have the same probabilty. If the distributions of equations 4.6 or 4.7 are taken for the distribution of the bit strings it is immediately obvious that because effectively the bits are treated by these distributions as independent (a large unobserved portion is assumed) that the entropy will simply be the entropy for a single bit times the number of observed bits. Thus, letting q be the probability that a single bit is one we have for the large string case
equation1304
which is trivially verified for the binomial form tex2html_wrap_inline13835. For the Poisson form, note that effectively we have assumed that tex2html_wrap_inline13837 while m q is constant. The argument is slightly trickier and the limit of m must be taken carefully, but the result is again that the entropy per bit is -q log(q) - (1-q) log(1-q).

Clearly, if the entropy is to show the effects of the finite unobserved portion then we must compute with the non-limiting case form of the distribution, equation 4.3.



David Wolf
Tue Mar 25 08:11:49 CST 1997