Neural Memory Update with Surprise and Momentum
Backpropagation, Training & Optimization DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding Dynamic Neural Memory and Adaptive Optimization, Surprise-based gating, Momentum accumulation, Recursive state updates, L2 norm error calculation, Hyperparameter tuning (alpha/beta), Deep Learning Foundations, Optimization Algorithms, Dynamical Systems, Stochastic Processes, Signal Processing, Backpropagation through time, Adaptive Learning Rates, Exponential Moving Averages, Gradient Descent Variants, State Space Modeling.
Implement a 'NeuralMemory' class that maintains a state vector. The update rule for the state S at time t is defined as: S t = (1 alpha) S {t 1} + alpha (input t surprise factor) + beta momentum t, where surprise factor is the L2 norm of the difference between input and current state, and momentum t is the exponential moving average of previous changes.