Recursive Language Models and Advanced LLM Architectures

“This course contains the use of artificial intelligence.”

New architecture papers arrive every week, and most claims are made on hardware and at scales you will never reproduce. This course teaches you to understand the main ideas beyond the plain transformer, implement small versions of them yourself, and measure them fairly on an ordinary CPU, so you can decide what is worth using.

You start by building your own bench and lab environment: CPU PyTorch, Hugging Face transformers and a pinned set of dependencies. Then you build a tiny decoder-only transformer from scratch, train it, and record its speed, memory and quality in a results table. One benchmark harness is reused by every architecture that follows, so every row in the table is comparable.

Recursion is covered in all three of its meanings. You build recursive networks over trees and a Tree-LSTM, then weight-tied looped transformers, and look at Universal Transformers, Mixture-of-Recursions, HRM and the Tiny Recursive Model. The Recursive Language Model section builds an RLM loop by hand: the long prompt lives as a variable in a Python REPL, and the root model calls sub-models on pieces of it. You run it against a local model served by Ollama, read the full trace, try the official library, and see where a small root model fails.

The middle of the course works through the main efficiency ideas. Sparse, sliding-window and linear attention are implemented and timed; FlashAttention, MQA, GQA and latent attention are explained with honest notes on what needs a GPU. You implement a selective state space layer, train a tiny Mamba-style model, and compare pretrained Mamba and Pythia on the same data. RWKV, RetNet, Gated DeltaNet and today's hybrid stacks are explained, and an associative recall task shows why hybrids keep an attention layer.

Mixture of experts gets its own section: you build a router, watch expert load with and without the balancing loss, cause router collapse on purpose, and inspect the routing choices of a real small MoE model token by token. Memory follows, from external memory networks to the KV cache, which you measure, size from the model configuration and manage with the transformers cache classes and attention sinks.

Retrieval is treated as external memory you can update. You compare chunking strategies, choose a small embedding model and check its licence, build FAISS indexes, combine BM25 and dense search, add a cross-encoder re-ranker, and score retrieval with recall, MRR and nDCG on a labelled set. A complete local RAG pipeline answers with citations and a faithfulness check, and is compared with an RLM and a long prompt on the same questions.

The last sections are about decisions. You learn the rules of fair comparison, the pitfalls of perplexity, how to measure throughput and time to first token properly, and what a toy experiment can and cannot say about scaling. The capstone compares three architectures on one task with one harness and ends with a written architecture selection report. Eleven downloadable documents come with the course, including a paper-claim checklist, the fair comparison protocol, a CPU bench setup guide, KV cache sizing and MoE checklists, a RAG evaluation template and the selection report template. All numbers in the course are measured on the course bench and are presented as such.