Entropy (disorder) is badHomogeneity is good

• Let S be a set of examples

• Entropy(S) = -P log2(P) - N log2(N)

  • where P is proportion of pos example
  • and N is proportion of neg examples
  • and 0 log 0 == 0

• Example: S has 9 pos and 5 neg

  Entropy([9+, 5-]) = -(9/14) log2(9/14) - (5/14)log2(5/14)
  
  = 0.940

Previous slide

Next slide

Back to first slide

View graphic version