Mutual Information
Probability Theory DS practice problem on Onlearn.
Difficulty: medium.
Topics: Mutual Information, Kullback-Leibler Divergence, Shannon Entropy, Joint Probability Distribution, Conditional Mutual Information, Pointwise Mutual Information, Information Theory, Probability Theory, Statistical Inference, Machine Learning Theory, Signal Processing, Entropy Measures, Dependency Analysis, Distribution Modeling, Feature Selection, Channel Capacity.
Compute the mutual information between two random variables X and Y given their joint probability distribution. Mutual information measures how much information one variable provides about another it quantifies the reduction in uncertainty about one variable given knowledge of the other. The result is 0 when variables are independent and maximized when they are perfectly dependent.