Entropy & Cross-Entropy
Probability Theory DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding Entropy and Cross-Entropy in Information Theory, Base-2 Logarithm, Vectorized Probability Operations, Numerical Stability in Logarithms, Gibbs Inequality, Maximum Likelihood Estimation, Probability Theory, Information Theory, Statistical Learning, Optimization Theory, Numerical Analysis, Shannon Entropy, Kullback-Leibler Divergence, Probability Mass Functions, Logarithmic Loss, Information Gain.
Implement two functions: 'calculate entropy(probs)' to compute the Shannon entropy of a probability distribution, and 'calculate cross entropy(p, q)' to compute the cross entropy between two probability distributions 'p' (true) and 'q' (predicted). Handle the edge case where probability is 0 by using a small epsilon or filtering.