RMSProp Optimizer
Backpropagation, Training & Optimization DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding RMSProp Optimization Dynamics, Exponentially Weighted Moving Average (EWMA), Second Moment Estimation, Gradient Normalization, Epsilon Smoothing, Non-stationary Objective Functions, Optimization Algorithms, Gradient-Based Learning, Numerical Analysis, Deep Learning Foundations, Stochastic Calculus, Adaptive Learning Rates, Moving Averages, Loss Landscape Geometry, Parameter Regularization, Convergence Stability.
Implement an RMSProp optimizer step function. Given a parameter vector, its gradient, the current moving average of squared gradients (cache), a decay rate (gamma), a learning rate (lr), and a smoothing epsilon, calculate the updated parameter and the updated cache.