Prompt Engineering, RAG or Fine-Tuning? Decision Playbook

Decide between prompting, retrieval, and fine-tuning — then evaluate cost, quality, security, and readiness.

New

Created byDr. Amar Massoud

Last updated 8/2026

English

What you'll learn

Classify any LLM use case as knowledge-bound, behaviour-bound, or both, using a documented seven-question decision tree
Decide when prompt engineering alone is genuinely sufficient, and when it is a false economy
Specify a production RAG pipeline: chunking, embeddings, hybrid retrieval, reranking, grounding, and citation policy
Assess a dataset and state whether supervised fine-tuning is viable, premature, or simply the wrong tool
Select and defend a reference architecture that combines techniques, including routing, fallback, and escalation
Build a cost-per-1,000-requests model and a latency budget for competing designs, and find the crossover point
Produce a threat model and a production-readiness sign-off pack for an LLM application
Write an Architecture Decision Record that survives review by an architect, a CISO, and a CFO

Course content

9 sections • 56 lectures • 4h 55m total length

Welcome — The Question Every Enterprise LLM Project Hits First5:23
What You'll Be Able to Do by the End4:57
How This Course Is Structured4:56
Meet Vantia Insurance Group3:52
Section 1 Knowledge Check

Where Prompting Is Genuinely Sufficient5:28
Structured Prompts — Role, Task, Constraints, Output Contract4:06
Few-Shot Examples — When They Help and When They Hurt5:02
Context Management and the Cost of Long Prompts5:01
Prompt Versioning, Testing, and Regression Suites5:12
Prompt Injection and Data Leakage Risks5:34
Lab — A Prompt-Only Claims Summarizer for Vantia4:59
Section 3 Knowledge Check
Build a Versioned Prompt Contract With a Frozen Evaluation Set

What RAG Actually Solves — and What It Doesn't5:01
The RAG Pipeline End to End5:16
Chunking Strategies That Survive Contact With Real Documents4:56
Embeddings and Vector Databases — The Selection Criteria5:01
Hybrid Retrieval and Reranking5:05
Grounding, Citations, and Refusal Behaviour3:44
Evaluating RAG — Retrieval Metrics vs Generation Metrics5:02
Lab — A RAG Assistant Over Vantia's Policy Wordings3:28
Section 4 Knowledge Check
Specify a Retrieval Pipeline and Measure Its Retrieval Half

What Fine-Tuning Changes — and What It Cannot Fix3:41
Supervised Fine-Tuning — The Mechanics Without the Mathematics4:48
LoRA, QLoRA, and PEFT — Parameter-Efficient Adaptation4:41
Dataset Preparation — The 80% of the Work5:07
Dataset Readiness — Volume, Quality, Licensing, and Drift5:25
When Fine-Tuning Is the Wrong Answer5:05
Colab Lab — A Lightweight LoRA Run on Vantia's Triage Data4:46
Section 5 Knowledge Check
Run a LoRA Fine-Tune and Audit One of Your Own Datasets

Build the Evaluation Set Before You Build the System5:19
Accuracy, Faithfulness, and Hallucination Rate5:35
LLM-as-Judge — Where It Works and Where It Misleads5:28
Unit Economics — Cost Per Thousand Requests7:05
Latency Budgets and the Perceived-Speed Trade-Off5:35
Lab — Scoring Vantia's Three Options Side by Side7:00
Section 7 Knowledge Check
Model Cost Per Thousand Requests and Find the Crossover

Threat Modelling an LLM Application7:01
Prompt Injection, Tool Abuse, and Insecure Output Handling5:26
Data Privacy — What Leaves Your Boundary in Each Approach5:43
Data Residency, Retention, and Vendor Terms5:30
Model Risk Management and Regulatory Expectations5:30
Monitoring, Drift, and Incident Response in Production5:29
Lab — Threat Model and Production-Readiness Review for Vantia6:52
Section 8 Knowledge Check
Threat-Model One LLM Feature and Complete a Readiness Review

Requirements

Working familiarity with what an LLM is and how an API call works
Comfort reading an architecture diagram
No machine learning, maths, or fine-tuning experience required
Optional, for the Colab lab only: basic Python literacy, as the notebook runs end to end without editing

Description

This course contains the use of artificial intelligence.

Most enterprise LLM projects stall on the same question: do we prompt, retrieve, or fine-tune? Pick wrong and you spend six months and a large budget solving a problem the cheapest lever would have solved in three weeks.

This is not a twenty-hour AI-engineering bootcamp. It is a decision playbook. The deliverable is judgment — the ability to look at a use case and defend a customization decision in front of an architecture review board, a CISO, and a CFO.

What makes this course different

Decision-first, not tool-first — every section ends by routing a real use case, not by finishing a tutorial
Enterprise constraints are first-class — cost, latency, privacy, data residency, model risk, and monitoring get real coverage, not an afterthought
One model company runs the whole course — you follow a single specialty insurer through three real use cases and watch one architecture evolve, rather than nine disconnected demos
Every section ships a reusable artefact — a decision matrix, an ADR template, a dataset-readiness checklist, a RAG evaluation workbook, a cost calculator, a threat-model template, and a production-readiness checklist

What you will actually do

Route use cases through a documented seven-question decision tree
Write and version a structured prompt with a frozen evaluation set
Specify a production retrieval pipeline and measure its retrieval half separately from its generation half
Run a lightweight LoRA fine-tune in Colab and audit a dataset for readiness
Model cost per thousand requests and find the volume where the ordering flips
Threat-model an LLM feature and complete a production-readiness review
Write an Architecture Decision Record recommending one approach, with rejected alternatives and their rationale

You need to know what an LLM is and how an API call works. You do not need a maths background, an ML background, or any fine-tuning experience.

Who this course is for:

AI solution architects designing LLM systems for enterprise use
Technical product managers scoping LLM features and defending the budget
Enterprise developers evaluating prompting, retrieval, and fine-tuning options
Consultants advising clients on practical AI adoption
IT and security leaders governing LLM deployments
Teams that have to decide how to customize an LLM and justify the choice

Prompt Engineering, RAG or Fine-Tuning? Decision Playbook

What you'll learn

Explore related topics

Course content

Course Introduction and the Decision Problem4 lectures • 19min

The LLM Customization Decision Framework6 lectures • 33min

Prompt Engineering as an Enterprise Discipline7 lectures • 35min

Production RAG8 lectures • 38min

Fine-Tuning7 lectures • 34min

Combining the Techniques6 lectures • 29min

Evaluating Quality, Cost, and Latency6 lectures • 36min

Security, Privacy, and Governance7 lectures • 42min

Capstone and Close5 lectures • 30min

Requirements

Description

Who this course is for: