20
Entropy
Given a set of training vectors S, if there are c
classes,
Entropy(S) = å -pi log
(pi)
Where pi is the proportion of category i examples
in S.
i=1
c
2
If all examples belong to the same category, the
entropy
is 0.
If the examples are equally mixed (1/c examples of
each
class), the entropy is a maximum at 1.0.
e.g. for c=2, -.5 log .5 - .5 log .5 =
-.5(-1) -.5(-1) = 1
2
2