Shannon Entropy

Describes how much information we need on average to encode an outcome of a given distribution formed from Random Variable $X$

• Allows calculating uncertainty of distribution of RV $X$

Formula

$$H(X)=-\sum _{x\in \mathcal{X}}P(x){\log }_{b}P(x)$$