Reading List

Vaswani et al. · The Transformer paper. Know every component cold.

Kaplan et al. (OpenAI) · Defines how model size, data, compute relate to loss.

Hoffmann et al. (DeepMind) · Corrects scaling law - 20 tokens per parameter optimal.

He et al. (MSRA) · ResNets. Why skip connections enable very deep networks.

Krizhevsky et al. · Started deep learning revolution. Know the architecture.

Karpathy · Intuition building. Read before understanding LSTMs.

Olah (colah) · Best visual explanation of LSTMs. Required reading.

Dosovitskiy et al. · Transformers for vision - architecture and pretraining insights.

Dao et al. · IO-aware attention. How to actually implement Transformers efficiently.

Hu et al. · Parameter-efficient fine-tuning. Ubiquitous in practice.

Ouyang et al. (OpenAI) · Original RLHF paper. How ChatGPT was made.

Rafailov et al. · Simpler RLHF alternative. Used in most modern models.

Bai et al. (Anthropic) · How Claude was trained. Key for Anthropic interviews.

Radford et al. (OpenAI) · Contrastive multimodal learning foundation. Zero-shot vision.

Radford et al. (OpenAI) · Encoder-decoder Transformer for speech. Audio chunk processing.

Kirillov et al. (Meta) · Promptable segmentation. Key CV model of 2023.

Ho et al. · Diffusion model foundations. Forward/reverse process.

Brown et al. (OpenAI) · In-context learning, scaling. The paper that changed everything.

Touvron et al. (Meta) · Open source LLM architecture. GQA, RLHF at scale.

Huang et al. (Google) · Ilya's reading list. Pipeline parallelism foundations.

Graves et al. · Ilya's reading list. Memory + attention before Transformers.

Kingma & Welling · VAE foundations. ELBO, reparameterization trick.

Dettmers et al. · Fine-tune 65B model on single GPU. Critical for startup ML.

Jiang et al. · GQA + sliding window attention. Dense model efficiency.

DeepSeek AI · MLA attention, fine-grained MoE, GRPO training. Modern architecture.