Udemy
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Real-Time Data Engineering: Python, dbt & DuckDB
New
8 students

Real-Time Data Engineering: Python, dbt & DuckDB

Build an automated API data pipeline with Bronze, Silver & Gold layers, dbt transformations, tests and Streamlit
Created byBluelime Learning Solutions
Last updated 9/2026
English

What you'll learn

  • Build an end-to-end real-time data engineering pipeline using Python, DuckDB, dbt, and Streamlit.
  • Extract live cryptocurrency market data from a REST API using Python.
  • Use the Python Requests library to retrieve and process JSON data from web APIs.
  • Transform API responses into structured datasets using Pandas DataFrames.
  • Implement incremental data loading to continuously capture new API data.
  • Store and query analytical data efficiently using DuckDB.
  • Design a Bronze, Silver, and Gold data architecture for a modern analytics pipeline.
  • Build a Bronze Layer for storing raw historical API data
  • Configure dbt to connect to and transform data stored in DuckDB
  • Create dbt staging models to clean, rename, standardize, and prepare raw data.
  • Build analytics-ready Gold Layer models using SQL and dbt.
  • Use SQL window functions to identify and select the latest records from incremental data.
  • Define dbt sources and model dependencies using source() and ref().
  • Implement automated data quality tests with dbt for unique, non-null, and reliable data.
  • Automate API ingestion, dbt transformations, and testing with a Python orchestration script.
  • Use Python to execute external dbt commands programmatically.
  • Schedule recurring pipeline executions to automatically collect and transform new data
  • Build an interactive monitoring dashboard using Streamlit.
  • Create interactive data visualizations with Plotly.
  • Connect Streamlit directly to a DuckDB analytics database and display Gold Layer data
  • Understand how ingestion, transformation, testing, orchestration, and visualization work together in a complete data pipeline.
  • Apply practical data engineering techniques through a portfolio-ready real-world project.

Course content

3 sections • 12 lectures • 1h 14m total length

  • Introduction4:05

Requirements

  • No previous data engineering experience is required — the course explains the pipeline step by step.
  • Basic Python knowledge is helpful, but advanced programming skills are not required.
  • Basic SQL knowledge is helpful for understanding the dbt transformations and queries.
  • A Windows, macOS, or Linux computer capable of running Python and the course tools.
  • Internet access for installing packages and retrieving live data from the API.
  • Python and Visual Studio Code — installation and setup are covered in the course.
  • No paid database or cloud account is required — we use DuckDB locally.
  • No previous experience with DuckDB, dbt, Streamlit, Plotly, or API ingestion is required.
  • A willingness to follow along and build the project hands-on is all you need.

Description

Build a complete real-time data engineering pipeline using Python, DuckDB, dbt, Streamlit, and live API data.

Data engineering is about much more than writing SQL or moving data from one place to another. Modern data engineers need to understand how data is collected, stored, transformed, tested, automated, and delivered to users.

In this hands-on course, you will build a practical end-to-end data engineering project from scratch.

We will use a live cryptocurrency market API as our data source, allowing you to work with continuously changing real-world data rather than static CSV files.

You will start by setting up your Python development environment and installing the tools required for the project.

Next, you will build a Python ingestion process that connects to a REST API, retrieves live market data, processes the JSON response with Pandas, and incrementally loads each new snapshot into DuckDB.

You will then implement a modern Bronze, Silver, and Gold architecture.

The Bronze Layer will preserve historical API data. Using dbt, you will transform this raw data into a clean and standardized Silver Layer before creating an analytics-ready Gold Layer containing the latest market information.

You will also learn how to define dbt sources, create model dependencies, use SQL window functions, and implement automated data quality tests.

But we won't stop at transformation.

You will create a Python orchestration script that brings the different stages together, automatically fetching new API data, loading it into DuckDB, running dbt transformations, validating the results, and repeating the pipeline on a schedule.

Finally, you will build an interactive Streamlit monitoring dashboard connected directly to the Gold Layer and use Plotly to visualize the latest market data.

By the end of the course, you will understand how the different components of a modern data pipeline work together:

Live REST API → Python → Pandas → DuckDB → Bronze → dbt Silver → dbt Gold → Data Quality Tests → Automation → Streamlit Dashboard

This course is designed for aspiring data engineers, analytics engineers, data analysts, Python developers, students, and career changers who want practical experience building modern data pipelines.

No previous data engineering experience is required. We build the project step by step, making it accessible to beginners while developing practical, portfolio-ready skills.



Who this course is for:

  • Aspiring Data Engineers who want practical experience building a complete, end-to-end data engineering project.
  • Beginners in Data Engineering who want to understand how APIs, databases, transformations, automation, and dashboards work together
  • Python Developers who want to expand their skills into data engineering and analytics engineering.
  • Data Analysts who want to move beyond analysis and learn how data is collected, transformed, tested, and prepared for reporting.
  • Aspiring Analytics Engineers who want hands-on experience with SQL, dbt, data modelling, and data quality testing.
  • SQL learners who want to apply their skills to a practical modern data pipeline.
  • Students and career changers looking to develop job-relevant data engineering skills through a portfolio-ready project.
  • Developers interested in APIs who want to learn how to ingest live REST API data and build historical datasets.
  • Learners interested in modern data stacks who want practical experience with Python, DuckDB, dbt, Pandas, Streamlit, and Plotly.
  • Anyone who learns best by building and wants to create a working pipeline rather than only studying data engineering theory.

Report abuse