Embedding Quantization Quality via Cosine Similarity
MoE, Compression & Scaling DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding Embedding Quantization Quality via Cosine Similarity, Cosine Similarity, L2 Norm, Floating Point Precision, Dot Product, Batch Processing, Linear Algebra, Information Theory, Numerical Analysis, Vector Spaces, Optimization, Embedding Compression, Quantization Error, Distance Metrics, Vector Normalization, Dimensionality Reduction.
Implement a function measure quantization loss that takes two matrices of embeddings: original (float32) and quantized (e.g., int8 or float16). The function should return the average cosine similarity across all corresponding embedding pairs. Handle potential division by zero for zero vectors.