Next:
Up:
Previous:

Entropy(S) = expected number of bits needed to encode class (
or
)
of randomly drawn member of S (under the optimal, shortestlength
code)
Why?
 Information theory: optimal length code assigns
bits to
message having probability p.
So, expected number of bits to encode
or
of random member
of S: