Building LLMs from Scratch
Following Sebastian Raschka's comprehensive guide with insights from music technology and MIR
Learning Progress
Understanding LLMs
Introduction to large language models and transformer architecture
Working with Text Data
Tokenization, embeddings, and data preprocessing
Attention Mechanisms
Self-attention, multi-head attention, and causal masking
Implementing GPT
Building a GPT model from scratch in PyTorch
Pretraining
Pretraining on unlabeled data at scale
Classification Finetuning
Finetuning for downstream classification tasks
Instruction Following
Finetuning models to follow instructions
Published Content
Deep dives, experiments, and connections to music technology
Journey Starting Soon
I'm just getting started with this learning journey. Check back soon for detailed notebooks and insights!
About This Learning Journey
I'm working through "Build a Large Language Model (From Scratch)" by Sebastian Raschka, documenting my learning process with a unique perspective from my background in music technology and MIR.
What makes this series different:
- 🎵 Music Tech Connections - Drawing parallels between LLMs and audio processing
- 🔬 Deep Experiments - Going beyond the book with additional explorations
- 📊 Visualizations - Interactive plots and diagrams to build intuition
- 💭 Honest Reflections - Documenting challenges and "aha" moments
All code, experiments, and detailed notes are available in my GitHub repository.