SMOL-GPT icon
SMOL-GPT icon

SMOL-GPT

A minimal PyTorch implementation for training your own small LLM from scratch. Designed for educational purposes and simplicity, featuring efficient training, flash attention, and modern sampling techniques.

Cost / License

  • Free
  • Open Source (MIT)

Platforms

  • Python
  • Windows
  • Mac
  • Linux
  • BSD
  • PyTorch
SMOL-GPT alternatives  
2likes
9alternatives
0articles

Features

SMOL-GPT News & Activities

Highlights All activities

Recent activities

SMOL-GPT information

AlternativeTo Category

AI Tools & Services

GitHub repository

  •  1,473 Stars
  •  125 Forks
  •  9 Open Issues
  •   Updated  
View on GitHub

Popular alternatives

View all
SMOL-GPT was added to AlternativeTo by Paul on and this page was last updated .
No comments or reviews, maybe you want to be first?

Official Links

AppStores & Other Links

What is SMOL-GPT?

A minimal PyTorch implementation for training your own small LLM from scratch. Designed for educational purposes and simplicity, featuring efficient training, flash attention, and modern sampling techniques.

Features:

  • Minimal Codebase: Pure PyTorch implementation with no abstraction overhead
  • Modern Architecture: GPT model with:
  • Flash Attention (when available)
  • RMSNorm and SwiGLU
  • Efficient top-k/p/min-p sampling
  • Rotary embeddings - RoPE (Optional)
  • Training Features:
  • Mixed precision (bfloat16/float16)
  • Gradient accumulation
  • Learning rate decay with warmup
  • Weight decay & gradient clipping
  • Dataset Support: Built-in TinyStories dataset processing
  • Custom Tokenizer: SentencePiece tokenizer training integration