LLM Introduction Course (MI, University of Oxford)

Hands-on introduction to LLMs

A short four-week introduction to how modern decoder-only LLMs work end-to-end. Students will build a minimal GPT step by step and experiment with open-weight models to understand architecture design choices and post-training objectives. Lectures take an implementation-first approach and include guided coding exercises in PyTorch.

Learning outcomes

  • End-to-end understanding of decoder LLM training and inference.
  • Familiarity with architectural design choices in open models.
  • Understand inference and long-context bottlenecks in practice.
  • Run a minimal post-training pipeline (SFT + preferences).

Simon Vary

Email: simon.vary@stats.ox.ac.uk
Web: simonvary.github.io
Place: Mathematical Institute, Univ. of Oxford


Bring: laptop with Python + PyTorch.
Registration link: forms.gle/VQ7zM99qwAeQ8YRD9.

Schedule

Date Time / Room Lecture Materials
Wed 4th March 15:00–17:00
L4, MI		 1 — simpleGPT & basics
History, tokenizer, tensors, causal self-attention, training, metrics 		Slides • Code
Wed 11th March 15:00–17:00
L4, MI		 2 — Architecture design choices / open-weight models
Position encoding (RoPE, YaRN), attention variants (MQA/GQA/MLA), normalization (pre/post-norm) + activation choices
Wed 18th March 15:00–17:00
L6, MI (TBC)		 3 — Inference: KV-cache & long context
Prefill vs decode, KV-cache (compute vs memory), long-context bottlenecks, optional: in-context learning, mixture of experts (MoE)
Wed 25th March 15:00–17:00
L4, MI		 4 — Post-training: objectives, PEFT, and preferences
SFT, parameter-efficient fine-tuning (LoRA), preference learning (DPO), verifier-based rewards (RLVR), Chain-of-thought (CoT)

Lectures

Lecture 1 — simpleGPT: the end-to-end pipeline

Lecture 2 — Architecture design choices / open-weight models

Lecture 3 — Inference: KV-cache & long context

Lecture 4 — Post-training: objectives, PEFT, and preferences

References