GCP Data Engineering Project: MySQL to BigQuery via Dataflow

Build a real-world incremental pipeline with Cloud SQL, Dataflow, BigQuery, Cloud Composer, watermarks and MERGE logic.

New

Created byChandra Venkat

Last updated 7/2026

English

What you'll learn

Build an end-to-end incremental pipeline from Cloud SQL MySQL to BigQuery using Dataflow
Configure Cloud SQL, private networking, Dataflow, BigQuery, and Cloud Composer.
Implement watermark-based ingestion and BigQuery MERGE logic for new and updated records.
Orchestrate and monitor repeatable pipeline runs using Cloud Composer and Apache Airflow.

Course content

6 sections • 17 lectures • 4h 52m total length

Welcome to Data0to10:38
Create Your GCP Account5:00
Create a GCP Project and Set a Budget9:26

Install Visual Studio Code3:48
STEP 1: Install VS Code
Search: VS Code download
Download and install
During install:
Add to PATH
Add "Open with Code" options
Open VS Code once to verify
Install WSL and Ubuntu11:38
STEP 1: Install WSL
Open PowerShell as Administrator
Run:
wsl --install -d Ubuntu-22.04
Restart system if prompted
STEP 2: Setup Ubuntu
Open Ubuntu from Start menu
Create username: unix
Create password: unix
NOTE:
Password will NOT be visible while typing
STEP 3: Update System
Run:
sudo apt update
sudo apt upgrade -y
STEP 4: Install Python
Run:
sudo apt install python3 python3-pip -y
STEP 5: Verify Python Installation
Run:
python3 --version
STEP 6: Verify Python Location
Run:
which python3
Expected:
Python should be located at:
/usr/bin/python3
Connect Visual Studio Code to WSL3:50
STEP 1: Install Extensions in VS Code
Open VS Code
Install:
Python (Microsoft)
WSL (Microsoft)
STEP 2: Open Project in WSL
Open Ubuntu terminal
Run:
cd ~
mkdir -p sample
cd sample
To open VS Code in WSL:
code .
STEP 3: Verify WSL Connection
Terminal should look like:
unix@machine:~/sample$
STEP 4: Set Default Terminal (if needed)
In VS Code:
Ctrl + Shift + P
Search:
Terminal: Select Default Profile
Choose:
WSL / Ubuntu
STEP 5: Verify Python
Run:
python3 --version
which python3

Download the Project Files0:02
Open and Configure the Project in Visual Studio Code22:42
STEP 1: Prepare Working Directory
Open Ubuntu
Run:
cd ~
mkdir -p data0to1_pipelines
cd data0to1_pipelines

mkdir azure
cd azure
STEP 2: Copy Project Code
If your zip file is in Windows Downloads:
Unzip it
Copy the contents into the above WSL path using File Explorer
Then run:
code .
This will open VS Code from the azure (or respective cloud) folder.
STEP 3: Give Execute Permission
Run:
find . -type f -name "*.sh" -exec chmod 755 {} \;
STEP 4: Install Required Utilities
Run:
sudo apt install unzip dos2unix -y
STEP 5: Convert Shell Script Format
Run:
find . -type f -name "*.sh" -exec dos2unix {} \;
STEP 6: Going Forward – Open Project in VS Code
Run:
cd ~/data0to1_pipelines//
code .
Create the GCP Base Infrastructure(GCS, BigQuery, IAM)48:26
Monitor and Control GCP Costs3:44

Requirements

Basic familiarity with SQL and Python will help you get the most out of this course.
A computer with Internet Access
No prior Google Cloud, Dataflow, or Cloud Composer experience is required.

Description

Build a realistic GCP data engineering project that ingests retail data from Cloud SQL for MySQL into BigQuery using Google Cloud Dataflow.

You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, designing incremental loads, setting up Google Cloud resources, implementing the solution, validating the output, and orchestrating repeatable runs.

In this project, you will:

Prepare the source: Set up retail source tables in Cloud SQL for MySQL
Configure networking: Connect Cloud SQL and Dataflow using private networking
Build the ingestion pipeline: Move data from MySQL into BigQuery using Dataflow
Load data incrementally: Use watermarks to process new and updated records
Create BigQuery tables: Build staging and raw tables with partitioning and clustering
Merge incremental data: Use BigQuery MERGE procedures to update target tables
Orchestrate the pipeline: Deploy and schedule workflows using Cloud Composer and Apache Airflow
Validate the results: Query the loaded data and verify initial and incremental runs in BigQuery

More than isolated service demonstrations

You will build the project in your own Google Cloud account using the provided source code, sample retail data, configuration files, SQL scripts, and infrastructure setup scripts.

The focus is not only on getting the pipeline to run. You will understand why each Google Cloud service is used, how Cloud SQL, Dataflow, BigQuery, and Cloud Composer work together, and how the pipeline handles new and updated source records during subsequent runs.

By the end of the course, you will have a complete GCP data engineering project that you can practice, adapt for your portfolio, and explain clearly in interviews.

This course contains a promotion.

Who this course is for:

Aspiring data engineers who want to build a practical Google Cloud data engineering project.
Beginners and career switchers seeking hands-on experience with Dataflow, BigQuery, and Cloud Composer.
Working data professionals preparing for GCP data engineering projects and interviews.

GCP Data Engineering Project: MySQL to BigQuery via Dataflow

What you'll learn

Explore related topics

Course content

Getting Started3 lectures • 15min

Set Up Your Development Environment3 lectures • 19min

Understand the Business and Source Systems2 lectures • 20min

Set Up the GCP Project Environment4 lectures • 1hr 15min

Build the Incremental MySQL-to-BigQuery Project4 lectures • 2hr 42min

Continue your learning Journey1 lecture • 1min

Requirements

Description

Who this course is for: