Lasso Regression using ISTA
Linear Models DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding Proximal Gradient Descent for Sparse Optimization, Soft-Thresholding Operator, L1 Norm Penalty, Lipschitz Constant Estimation, Proximal Gradient Method, Convergence Criteria, Linear Algebra, Optimization Theory, Statistical Learning, Numerical Analysis, Convex Optimization, Gradient Descent, Proximal Operators, Regularization Techniques, Sparsity Induction, Eigenvalue Decomposition.
Implement Lasso Regression from scratch using the Iterative Soft Thresholding Algorithm (ISTA). Given a design matrix X (n samples, n features) and target vector y, implement a function that performs a fixed number of iterations to solve the objective: 1/(2n) ||y Xw||^2 + alpha ||w|| 1. Use a step size (learning rate) of 1/L, where L is the largest eigenvalue of X^T X. Assume the intercept is zero for simplicity.