Implementing Basic Autograd Operations
Backpropagation, Training & Optimization DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding Implementing Basic Autograd Operations, Chain Rule, Topological Sort, Jacobian-Vector Product, Directed Acyclic Graph, Gradient Accumulation, Calculus, Computational Graphs, Numerical Analysis, Object-Oriented Programming, Deep Learning Frameworks, Automatic Differentiation, Backpropagation Algorithms, Operator Overloading, Symbolic Differentiation, Activation Functions.
Special thanks to Andrej Karpathy for making a video about this, if you haven't already check out his videos on YouTube https://youtu.be/VMj 3S1tku0?si=gjlnFP4o3JRN9dTg. Write a Python class similar to the provided 'Value' class that implements the basic autograd operations: addition, multiplication, and ReLU activation. The class should handle scalar values and should correctly compute gradients for these operations through automatic differentiation.