KL Divergence

Its a metric which is used to measure the similarity between two probability distribution P and Q.

For probability distributions P and Q of a discrete random variable their K–L divergence is defined to be

$D_{\mathrm {KL} }(P\|Q)=\sum _{i}P(i)\log {\frac {P(i)}{Q(i)}}.\!$