Number Format Precision Comparison (FP16 vs BF16 vs FP8 vs FP4)
Backpropagation, Training & Optimization DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding Floating Point Precision and Numerical Stability in Deep Learning, IEEE 754 Standard, Brain Float 16 (BF16) structure, E4M3 vs E5M2 Formats, Subnormal Number Handling, Bit-level Quantization Error, Numerical Analysis, Computer Architecture, Deep Learning Optimization, Floating Point Arithmetic, Data Quantization, Dynamic Range vs Precision, Mantissa Truncation, Exponent Bias, Underflow and Overflow, Stochastic Rounding.
Implement a Python class 'PrecisionSimulator' that takes a list of float32 numbers and computes their representation in FP16 (IEEE 754), BF16 (Brain Float), FP8 (E4M3), and FP4 (E2M1). For each format, return the quantized value and the relative error. Assume standard bit widths: FP16 (1+5+10), BF16 (1+8+7), FP8 (1+4+3), FP4 (1+2+1).