Positional Encoding Calculator
Attention Mechanisms DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding Positional Encoding Calculator, Sinusoidal Positional Encoding, Half-Precision Floating Point, Broadcasting Rules, Input Validation Logic, Dimensionality Scaling, Deep Learning Architectures, Numerical Computing, Sequence Modeling, Signal Processing, Tensor Operations, Attention Mechanisms, Vectorization Techniques, Floating Point Precision, Input Embedding Layers, Trigonometric Transformations.
Write a Python function to implement the Positional Encoding layer for Transformers. The function should calculate positional encodings for a sequence length (position) and model dimensionality (d model) using sine and cosine functions as specified in the Transformer architecture. The function should return 1 if position is 0, or if d model is less than or equal to 0. The output should be a numpy array of type float16.