Regularization via Information Bottleneck
Initialization, Normalization & Regularization DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding the Information Bottleneck Principle in Deep Learning, Reparameterization Trick, Information Bottleneck Principle, Evidence Lower Bound (ELBO), Stochastic Latent Embeddings, Compression-Distortion Trade-off, Information Theory, Bayesian Deep Learning, Statistical Learning Theory, Optimization, Probabilistic Graphical Models, Mutual Information, Variational Inference, Latent Variable Models, KL Divergence, Representation Learning.
Implement a simplified Variational Information Bottleneck layer that computes the loss contribution of the latent space bottleneck. Given a batch of encoder outputs (mean and log variance) and a beta parameter, calculate the KL divergence between the latent distribution and a standard normal prior N(0, I).