Show HN: GPT-2 124M trained from scratch – 2.48B tokens, 2.8h on 4 A40s
Category: ai-ml
Tags: gpt-2, pytorch, distributed-training, llm, education
Score: 5.5/10 (Innovation: 3, Technical: 7, Documentation: 8, Utility: 4)
A from-scratch PyTorch implementation of GPT-2 124M training, based on Karpathy's build-nanogpt, with explicit modules for model, data, distributed training, and evaluation. It demonstrates a full multi-GPU training pipeline with crash-safe checkpointing, HellaSwag evaluation, and reproducible results, making it a solid educational reference for understanding LLM training mechanics.
Target audience: ML engineers and researchers learning LLM training internals
Repository: https://github.com/upul/Build-GPT-2 · Python · MIT · 2 stars
View on Hacker News