Building Multimodal AI with RAG & Context Engineering

This course contains the use of artificial intelligence

Disclosure: AI tools were used only to assist in creating the course outline and course thumbnail. All instructional content, explanations, and project walkthroughs were fully created manually by the instructor.

Welcome to Building Multimodal AI with RAG & Context Engineering course. This is a comprehensive project based course where you will learn how to create and design a multimodal AI system that is capable of processing different types of inputs such as text, images, and audio files. This course is a perfect combination between Python and multimodal AI, making it an ideal opportunity to practice your programming skills while improving your technical knowledge in artificial intelligence development. In the introduction session, you will learn the basic fundamentals of multimodal AI, such as getting to know how it works, how Retrieval Augmented Generation is used to provide multimodal AI systems with access to relevant knowledge and information beyond their training data, and how context engineering is needed to organize and structure information before it is provided to a multimodal AI system. Then, in the next section, you will learn how to connect your system with large language model API using OpenRouter, Mistral AI API, and Google AI Studio, specifically, you will learn how to create API key and integrate API key, enabling your system to interact with AI and generate responses, summaries, insights, recommendations, and other intelligent outputs based on user inputs. Afterward, we will learn how to incorporate a vector database into our system, in this case, we will use Chroma to store and retrieve embeddings generated from text, images, and other data sources. By doing so, we enable the large language model to access relevant information from an external knowledge base, helping it generate more informed and context aware responses. Before starting the projects, we will learn about basic RAG concepts such as performing semantic search, metadata filtering, and multi query retrieval. This practice session will help us understand how information is retrieved from a vector database and provide a solid foundation for building a more intelligent and effective multimodal AI system. Then, in the next section, we will start the project. In the first project, we are going to build a Meeting Intelligence Multimodal AI Assistant that is capable of processing meeting related information from text documents, presentation slides, and audio recordings to generate meeting summaries, identify key discussion points, extract action items, and provide useful insights from the meeting. In the second project, we are going to build a real estate property evaluation multimodal AI assistant. This system will be able to analyze property descriptions, property images, and real estate agent narrations to evaluate properties, generate property assessment reports, identify strengths and weaknesses, and provide valuable insights. In the third project, we are going to build food quality checker multimodal AI assistant which is able to analyze food quality standards, food product images, and inspector observations to assess food quality, identify potential issues, evaluate compliance with quality requirements, and generate food inspection reports with recommendations for improvement. Lastly, at the end of the course, we will conduct functional and performance testing on our multimodal AI assistants. The objective is to make sure the systems can process multimodal inputs correctly, retrieve relevant information effectively, generate reliable outputs, and perform as expected under different usage scenarios.

Firstly, before getting into the course, we need to ask this question to ourselves? Why should we build multimodal AI assistants? Well, here is my answer, multimodal AI assistants can process and analyze multiple types of data, such as text, images, and audio, allowing them to understand situations more comprehensively than conventional AI systems. As a result, they can automate complex tasks, improve operational efficiency, and deliver more useful insights.

Below are things that you can expect to learn from this course:

• Learn the basic fundamentals of multimodal AI, RAG, and context engineering

• Learn how to connect your system to LLM API like Mistral, Gemini, Open Router, Groq, and Github

• Learn how to connect LLM to Pinecone vector database

• Learn how to connect LLM to Chroma vector database

• Learn how to perform basic semantic search

• Learn how to perform metadata filtering

• Learn how to conduct multi query retrieval

• Learn how to build meeting intelligence multimodal AI assistant

• Learn how to build real estate property valuation multimodal AI assistant

• Learn how to build food quality inspection multimodal AI assistant

• Learn how to build and design simple user interface using Gradio