(English)RAG with Local LLMs: Ollama, LangChain & ChromaDB

Define a working setup: a local generation response and an embedding vector returned from Python. These two checks will confirm that the core services are ready.

Install Ollama and understand how its background service works. Check the installed version and learn what to inspect when the local endpoint is unavailable.

Download the local generation model and send a first English prompt. Compare model sizes and the memory trade-offs involved.

Set up the model that converts text into vectors. Distinguish the role of an embedding model from the role of a text-generation model.

Create a project folder, activate an isolated Python environment and install the supplied dependencies. Use the downloadable commands to avoid transcription errors.

Configure VS Code and Jupyter to use your project environment. Diagnose import failures caused by selecting a different Python interpreter.

Call both local models from Python. Inspect the generated response and embedding dimensions before moving on to document retrieval.

Set the goals for working with embeddings and similarity. You will connect the mathematical representation to practical document search.

Build an intuition for sentence embeddings and vector dimensions. Understand why meaning can be compared numerically without assigning a human-readable meaning to each coordinate.

Generate a vector from text and examine its values, dimension and norm. Understand what normalization changes and what it does not.

Calculate similarity from dot products and vector norms. Compare several sentences and interpret a similarity matrix.

Embed a small corpus, score it against a question and sort the results. This hands-on exercise exposes the core operation behind vector retrieval.

Compare bge-m3 and nomic-embed-text using the same English questions and candidate passages. See why an easy two-choice task may hide differences that appear in a larger candidate set.

Examine identifiers, negation, numerical details and paraphrases. Use concrete failure cases to understand when semantic similarity does not identify the required evidence.

Review vectorization, cosine similarity and model comparison. Connect these foundations to the document preparation decisions in the next section.

Define the path from source documents to a searchable vector database. Plan extraction, cleaning, chunking and metadata together.

Locate the fictional corporate documents and understand their usage conditions. Distinguish supplied course data from third-party material that must be downloaded and handled under its own license.

Extract text and inspect the reading order instead of trusting a successful parse. Compare a regular PDF with a two-column example and remove structural HTML noise.

Apply practical text-cleaning rules and compare the same passage before and after. Check that important section numbers and policy content survive the cleanup.

Compare fixed-size splitting with semantic document boundaries. Consider chunk length, overlap and whether a passage contains enough context to support an answer.

Configure a recursive text splitter and inspect chunk boundaries. Adapt separators to English text and understand the interaction between size and overlap.

Change chunk size while keeping the task comparable. Inspect retrieval results and use evidence to choose a setting rather than assuming one size is always best.

Store document names, article labels and update information alongside each chunk. Build the metadata needed to display and trace an answer's sources.

Create a persistent collection, store embeddings and query it. Understand the difference between vector distance and similarity and keep the embedding model consistent.

Plan a complete RAG pipeline. Define the roles of the retriever, prompt, local language model and output parser.

Compare direct model calls with a framework-based pipeline. Learn which abstractions are useful and when simpler code is easier to understand.

Turn the vector store into a retriever and choose how many passages to return. Inspect retrieved documents before introducing answer generation.

Write a prompt that uses retrieved evidence, identifies sources and handles missing information. Understand why refusal instructions can also make a model decline answerable questions.

Compose retrieval, prompt formatting, local generation and parsing. Trace the values passed between components so the chain remains debuggable.

Ask questions against the supplied documents and inspect the responses. Confirm that retrieval and generation work together before measuring quality.

Expose the retrieved document and article information alongside the answer. Help users verify the evidence rather than relying on a fluent response alone.

Try questions that stress identifiers, negation and evidence spread across documents. Use the observed failures to motivate systematic evaluation.

Define a repeatable improvement loop. Separate retrieval quality, answer quality and latency before choosing a technique.

Create questions, reference passages and answer checks across several difficulty categories. Understand the limits of keyword checks and the need to review individual answers.

Calculate retrieval coverage and ranking quality. Use examples and tests to distinguish finding all required evidence from placing the first relevant passage near the top.

Run the baseline against the English evaluation set and inspect category-level results. Keep the recorded run so later changes can be compared on the same task.

Provide reference evidence directly to the model as an oracle experiment. Use the result to distinguish missing evidence from failures to use available evidence.

Compare individual prompt rules and combinations on the same questions. Measure refusals and answer checks, inspect examples and avoid assuming that adding more instructions always helps.

Implement keyword retrieval and combine rankings with Reciprocal Rank Fusion. Compare the hybrid result with vector retrieval before deciding which approach to keep.

Add a cross-encoder reranker and evaluate its cost and benefit. Check whether better passage rankings translate into better final answers.

Review the recorded experiments side by side. Balance retrieval metrics, answer checks and latency, and document why a particular configuration was selected.

Review the application you will assemble from the evaluated components. Connect document ingestion, answers, sources and the local user interface.

Set the use case and document scope using the fictional company policies. Define answerable and unanswerable questions and locate the completed project.

Inspect the five supplied documents and their article structure. Confirm the dataset and metadata before building the application index.

Move loading, chunking and indexing into a command-line workflow. Make the process repeatable and inspect what was added to the collection.

Track changed, added and removed documents with a manifest. Understand why obsolete chunks must be removed when a document changes.

Check the application prompt against the same evaluation set. Preserve the selected prompt and rerun checks when code or documents change.

Test questions that the documents cannot answer. Examine whether score thresholds reject irrelevant questions without also rejecting valid ones.

Rewrite follow-up questions into standalone retrieval queries. Show the rewritten question so mistakes remain visible and debuggable.

Connect the RAG pipeline to a local chat interface with source display and feedback. Keep the application bound to the local machine.

Ingest a separate set of documents and repeat the checks. Address extraction problems and protect confidential data while troubleshooting.

Distinguish incremental document updates from changes that require a full rebuild. Check embedding-model consistency before querying an existing collection.

Consider document volume, concurrency, latency and operational complexity. Identify when the simple local architecture needs additional components.

Survey query expansion, multi-stage retrieval and other extensions. Choose experiments based on diagnosed failure modes rather than technique names.

Understand how retrieved knowledge and adapted model behavior can complement each other. Identify what must be evaluated when combining the two approaches.

Balance memory, model size, response time and quality. Use your own workload measurements to guide hardware and model choices.

Review the full workflow and choose a next project. Carry forward the habit of measuring changes, checking evidence and protecting the data you process.