TinyGPT

TinyGPT: Decoder-Only Transformer Language Model

Project Overview

I built a decoder-only Transformer language model (TinyGPT) from scratch in PyTorch without relying on high-level NLP libraries like Hugging Face Transformers. The goal of the project was to understand the internal workings of modern Large Language Models by implementing every major component myself.

Core Features Implemented a GPT-style decoder-only Transformer architecture Built a complete autoregressive text generation pipeline Implemented causal self-attention so the model only attends to previous tokens Added multi-head attention for learning different contextual relationships simultaneously Implemented learned token embeddings and positional embeddings Used residual connections and Layer Normalization for stable training Built the feed-forward network used inside each Transformer block Implemented next-token prediction using autoregressive decoding Trained the model using Cross-Entropy Loss and optimized it with AdamW Implemented model checkpointing to save and reload trained weights Technical Stack Python PyTorch Neural Networks Transformers Self-Attention Natural Language Processing (NLP) Dataset

The model was trained on the text of The Adventures of Sherlock Holmes, learning statistical patterns in natural language and generating text in a similar style.

What I Learned

This project helped me gain a practical understanding of:

Tokenization Embedding layers Positional encoding Query, Key, and Value projections Multi-head attention Attention masking Residual connections Layer Normalization Feed-forward networks Language modeling Next-token prediction Sampling strategies Transformer architecture Challenges Solved

During development, I debugged and resolved issues related to:

Tensor shape mismatches Attention masking Position embeddings Sliding context windows during generation Model serialization and checkpoint loading Training stability Autoregressive inference

Built With

• dataset
• natural-language-processing
• python
• pytorch
• selfattention
• transformer