GPT-2 Text Generation
Text Generation & NLP Evaluation DS practice problem on Onlearn.
Difficulty: hard.
Topics: Understanding GPT-2 Text Generation, Positional Encoding, Multi-Head Self-Attention, Layer Normalization, Softmax Sampling, Feed-Forward Networks, Natural Language Processing, Deep Learning Architectures, Generative Modeling, Sequence Modeling, Model Inference, Transformer Blocks, Autoregressive Decoding, Embedding Layers, Attention Mechanisms, Normalization Techniques.
Implement a Simplified GPT 2 like Text Generation Function You are tasked with implementing a simplified GPT 2 like text generation function in Python. This function will incorporate the following components of a minimal GPT 2 architecture: Token Embeddings : Map input tokens to dense vector representations. Positional Embeddings : Add positional information to token embeddings. Multi head Attention : Attend to various parts of the sequence. Feed Forward Network : Process attention outputs through a dense layer. Layer Normalization : Stabilize the training process. The function must take in the following parameters: 1. Prompt : The initial text to guide the generation process. 2. Number of Tokens to Generate : Specify how many tokens to output. Your function should output the generated text. Additionally, utilize the helper function load encoder hparams and params to retrieve: A dummy encoder. Model hyperparameters. Model parameters. Build your text generation logic around these components. This exercise is designed to help you understand the core concepts behind GPT 2's autoregressive text generation.