Train OpenSource Large Language Models from Zero to Hero

Download the course's code resources attached to this section or upload them to your own GitHub, or request access by emailing me to be added to the GitHub.

Install transformers, datasets, and torch from Hugging Face; tokenize data with gpt2, create train and validation sets, move to gpu, train with the adam optimizer, and generate text for evaluation.

Explore free and paid GPU options to train language models, including Google Colab Pro and Paperspace, with tips on remote access via VSCode, Jupyter, and RunPod for scalable experiments.

Master supervised fine-tuning of large language models with LoRA and quantization to reduce memory usage while preserving performance, using freezing, paft, and efficient parameter updates.

Learn to train your first large language model in a fully supervised fashion, balancing memory, performance, and size while configuring data, tokenizer, and trainer for SFT.

freeze most model weights to save memory, train only the lm head, adjust batch size, and discuss the memory and performance trade-off.

Train an open-source llama 38b model with LoRA to achieve instruction tuning, configure chat templates and embedding tokens, and use quantization to boost training efficiency while evaluating improved instruction following.

Explore quantization and data types to save memory and train models with less gpu power, using nf4 4-bit quantization with 16-bit lower weights and svd-based ab initialization to compensate loss.

Train open-source large language models with QLoRA quantization, unfreeze embeddings, and fine-tune a llama 38B model while managing memory and evaluating training progress.

Explore Dora, a variation of Laura, that speeds up fine-tuning by updating only a small part of weights, enabling faster convergence via magnitude and direction decomposition using the L2 norm.

Use DoRA to improve stability in training open-source large language models, comparing Mistral 7B instruct with llama, detailing architecture, training setup, and magnitude vector optimization, plus future regularization.

Enhance open-source model training speed by applying flash attention, an I/O aware technique that minimizes data transfers across SRAM, HBM, and DRAM, boosting GPU efficiency.

Explore NEFTune, an embedding-level regularization technique that adds noise to token embeddings during fine-tuning, teaching large language models to ignore noise and train robust instruction tuning.

Apply flash attention with regularization to boost speed and robustness; configure Neptune noise alpha on the embedding layer and log with Wandb to improve validation loss for Mistral 70.

Learn how preference alignment boosts large language models through direct preference optimization (DPO), moving from pre-training to instruction tuning with a focus on chosen versus rejected outputs.

Demonstrate DPO training to align an open-source Mistral model to human preferences, using chosen and rejected prompts, SFT, dataset inspection, and practical beta and learning-rate guidance.

Discover how the KTO approach simplifies data curation for training LLMs by tagging any answer as good or bad, removing the need for chosen and rejected pairs.

Launch a keto Jupyter notebook to prepare data for fine-tuning an open-source model, balancing positive and negative labels with weights, and using Cato and Orpo to improve data curation.

Learn how Orpo trains language models end-to-end for preference alignment in one step, combining the SFT loss with the Orpo loss to reduce two-step training and hyperparameter sensitivity.

Install dependencies, load the dataset, and set up the base llama model with a chat template and tokenization for training; results show decreasing validation and a flat reward margin.

Explore multi-GPU training strategies for large language models, including distributed data parallel, model parallel, and sharded model parallel, using Deepspeed zero one, zero two, and zero three.

Deploy a multi-gpu node with two gpus using a VSCode server template, configure a device map for data-parallel sft training via accelerate cli, and speed up training.

Train a lama-based model with fully sharded data parallel across two gpus using the fsdp plugin and accelerate configuration to coordinate the optimizer and gradients.

Unsloth demonstrates faster fine-tuning by rewriting gradient computation with an open-source library, implements io aware training, patches Hugging Face trainer, and enables memory offloading and gradient checkpointing for context windows.

Launch onslaught training to speed up models and cut VRAM usage by monkeypatching the llama base, preparing the tokenizer, loading weights, and completing SftP training in 1 hour 20 minutes.