Adam Optimizer
Backpropagation, Training & Optimization DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding Adam Optimizer, Exponential Moving Average, Bias Correction, First Moment Estimation, Second Moment Estimation, Hyperparameter Tuning, Optimization Theory, Numerical Analysis, Calculus, Linear Algebra, Software Engineering, Adaptive Gradient Methods, Momentum-based Optimization, Stochastic Gradient Descent, Floating Point Arithmetic, Vectorized Computation.
Implement the Adam optimizer update step function. Your function should take the current parameter value, gradient, and moving averages as inputs, and return the updated parameter value and new moving averages. The function should also handle scalar and array inputs and include bias correction for the moving averages.