LoRA: Low-Rank Adaptation Forward Pass
MoE, Compression & Scaling DS practice problem on Onlearn.
Difficulty: medium.
Topics: LoRA: Low-Rank Adaptation Forward Pass, Singular Value Decomposition, Rank-Deficient Matrices, Adapter Injection, Gradient Scaling, Frobenius Norm, Linear Algebra, Deep Learning Theory, Parameter-Efficient Fine-Tuning, Computational Complexity, Transformer Architecture, Matrix Decompositions, Weight Update Dynamics, Low-Rank Approximation, Backpropagation Mechanics, Model Compression.
Implement the forward pass of LoRA (Low Rank Adaptation), a parameter efficient fine tuning technique for large language models. In LoRA, instead of updating all weights during fine tuning, we freeze the pretrained weights W and learn two small low rank matrices A and B that represent the weight update. Given an input x, frozen weights W, and LoRA matrices A and B with a scaling factor alpha, compute the output of the LoRA layer.