Entropy

• S is a sample of training examples
• $p_{\oplus}$ is the proportion of positive examples in S
• $p_{\ominus}$ is the proportion of negative examples in S
• Entropy measures the impurity of S
  
  $$Entropy(S) \equiv - p_{\oplus} \log_{2} p_{\oplus} - p_{\ominus} \log_{2} p_{\ominus}$$