Step-Size Selection for Tile Coding
Representation Learning, Advanced Theory & Miscellaneous DS practice problem on Onlearn.
Difficulty: medium.
Topics: Step-Size Selection for Tile Coding, Tile Coding, Learning Rate Decay, Hashing Collisions, Feature Vector Sparsity, Generalization Error, Reinforcement Learning, Numerical Optimization, Function Approximation, Statistical Learning Theory, Signal Processing, Linear Value Function Approximation, Stochastic Approximation, Sparse Feature Representations, Bias-Variance Tradeoff, Temporal Difference Learning.
Implement a function that performs a single value function update using tile coding with a properly adjusted step size. In tile coding, multiple overlapping tilings each activate exactly one tile for a given state. This means the total number of active features equals the number of tilings. If the step size is not adjusted to account for this, learning becomes unstable or overly slow. Your function should: 1. Accept a base learning rate, the number of tilings, a weight vector, a list of active tile indices for the current state, and a target value. 2. Compute an adjusted step size that accounts for the number of tilings. 3. Compute the current value estimate as the sum of weights at the active tile indices. 4. Update only the weights corresponding to active tiles using the adjusted step size and the prediction error (target minus current estimate). 5. Compute the value estimate after the update. Return a dictionary with keys: 'adjusted alpha' (the adjusted step size), 'prediction before' (value estimate before update), 'prediction after' (value estimate after update), and 'updated weights' (the full weight vector after update). All numerical values should be rounded to 4 decimal places.