User:Aravind V R/Formulas/Information theory

Entropy

Entropy is defined as

$\mathrm {H} (X)=\operatorname {E} [\operatorname {I} (X)]=\operatorname {E} [-\log(\mathrm {P} (X))]=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}$

Conditional entropy

$\mathrm {H} (X|Y)=-\sum _{i,j}p(x_{i},y_{j})\log {\frac {p(x_{i},y_{j})}{p(y_{j})}}$

Joint entropy $\mathrm {H} (X,Y)=-\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}P(x,y)\log _{2}[P(x,y)]$

Properties

Nonnegativity

$\mathrm {H} (X)\geq 0$

Subadditivity

$\max \left[\mathrm {H} (X),\mathrm {H} (Y)\right]\leq \mathrm {H} (X,Y)\leq \mathrm {H} (X)+\mathrm {H} (Y)$

Relation between entropies

$\mathrm {H} (Y,X)=\mathrm {H} (Y)+\mathrm {H} (X|Y)$

Mutual information

$\operatorname {I} (X;Y)=\sum _{y\in {\mathcal {Y}}}\sum _{x\in {\mathcal {X}}}{p(x,y)\log {\left({\frac {p(x,y)}{p(x)\,p(y)}}\right)}},$ for discrete random variables $X$ and $Y$

Conditional mutual information

${}\operatorname {I} (X;Y|Z)=\mathbb {E} _{Z}{\big (}\operatorname {I} (X;Y)|Z{\big )}={}\sum _{z\in {\mathcal {Z}}}\sum _{y\in {\mathcal {Y}}}\sum _{x\in {\mathcal {X}}}{p_{Z}(z)\,p_{X,Y|Z}(x,y|z)\log \left[{\frac {p_{X,Y|Z}(x,y|z)}{p_{X|Z}\,(x|z)p_{Y|Z}(y|z)}}\right]}={}\sum _{z\in {\mathcal {Z}}}\sum _{y\in {\mathcal {Y}}}\sum _{x\in {\mathcal {X}}}p_{X,Y,Z}(x,y,z)\log {\frac {p_{X,Y,Z}(x,y,z)p_{Z}(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}}.$

Properties

Nonnegativity

$\operatorname {I} (X;Y)\geq 0$

Symmetry

$\operatorname {I} (X;Y)=\operatorname {I} (Y;X)$

Relation to entropy

${\begin{aligned}\operatorname {I} (X;Y)&{}\equiv \mathrm {H} (X)-\mathrm {H} (X|Y)\\&{}\equiv \mathrm {H} (Y)-\mathrm {H} (Y|X)\\&{}\equiv \mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y)\\&{}\equiv \mathrm {H} (X,Y)-\mathrm {H} (X|Y)-\mathrm {H} (Y|X)\end{aligned}}$

Relation to Kullback–Leibler divergence

$\operatorname {I} (X;Y)=D_{\text{KL}}\left(p(x,y)\parallel p(x)p(y)\right).$