The GRPO Objective Function
RLHF, Reward Modeling & Human Feedback DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding Implement the GRPO Objective Function, Kullback-Leibler Divergence, Advantage Estimation, Clipping Mechanism, Likelihood Ratio, Policy Entropy Regularization, Reinforcement Learning, Optimization Theory, Probability and Statistics, Deep Learning Architectures, Alignment and Safety, Policy Gradient Methods, Constrained Optimization, Stochastic Processes, Transformer Decoder Models, Preference Modeling.
Implement the GRPO (Generalized Relative Policy Optimization) objective function used to optimize policy parameters in reinforcement learning. Your task is to compute the GRPO objective given the likelihood ratios, advantage estimates, old policy probabilities, reference policy probabilities, and apply the clipping mechanism and KL divergence penalty correctly to maintain training stability.