Fine-Tuning Fundamentals: Unlocking the Potential of LLMs

Explore the attention-based transformer architecture and compare GPT and BERT models, highlighting similarities, differences, and how textual inputs are processed.

Explore the transformer architecture with encoder and decoder stacks, self-attention, and multi-headed attention, and see how tokenization, BOS, separator tokens, padding, and context length shape inputs and outputs.

This lecture explains how transformer self-attention with multi-headed units processes text, contrasts it with RNNs/LSTMs, and highlights no information loss, parallel training, better long-range dependencies, and GPT and BERT models.

GPT models are auto-regressive transformers trained via pre-training on large text corpora and fine-tuned on task-specific data to excel at NLP tasks such as classification and summarization.

Explore how the BERT bidirectional encoder creates embeddings for text, using masking and sentence prediction, enabling downstream tasks like sentiment analysis and named-entity recognition.

Explore GPT-style decoder models, assess their suitability for conversations, and discuss drawbacks and practical fine-tuning approaches for real chat applications.

Explore how base models like gpt-2, gpt-3, llama are fine-tuned on conversations using instruction following datasets like alpaca and dolly to power chatgpt-style apps.

Load a base model in Colab, connect to T4 GPU, install transformers and accelerate, then fine-tune open lama 3 billion parameter model to improve instruction following and benchmark conversational capabilities.

Explore parameter efficient finetuning with low rank adaptation (LoRA) for task-specific tuning of large language models. Freeze the base model and train low-rank matrices to reduce trainable parameters and memory.

finetune a 3b open llama model in colab by preparing a 5000-point dataset, applying LoRA with specific parameters, and saving weights to google drive.

Explore inference and evaluation of a fine-tuned model, examining output quality, gibberish, and hallucinations, and how prompts influence explanations during model testing.

demonstrates how a bigger model (7B) trained on 52k data reduces hallucinations versus a 3B model, and guides fine-tuning toward a 13B model with full 52k data for improved outputs.

Explore prompt engineering, art and science of crafting inputs for finetuned models, to achieve coherent, accurate outputs. Learn why prompt engineers optimize prompts, especially for smaller models with limited instruction.

Explore how autoregressive models perform inference with beam search versus greedy search, revealing how beam size, top k, and top p shape output trajectories and text quality.

Replace deterministic greedy and beam search with sampling to promote stochastic exploration and richer text output. Use top k and top p to truncate and reweight probability distributions, shaping creativity.

Unlock the temperature parameter in large language models, a scaling factor that controls output probability sharpness and diversity from greedy choices at 0 to more diverse outputs above 1.

Identify key pitfalls in fine-tuning large language models, including repetitive outputs, hallucinations, uncertainty gaps, lack of output control, and slow inference, with deployment cautions for mission-critical applications.

Learn how to customize llms with instruction-following data, compare larger base models, address hallucinations with grounded generation, and apply reinforcement learning with human feedback for alignment.