A Real Banking Project on Google Cloud for Data Engineers

Name: A Real Banking Project on Google Cloud for Data Engineers
Rating: 4.3 (14 reviews)

Build Cloud SQL, GCS, Pub/Sub, BigQuery, Dataproc, PySpark and Airflow pipelines in a real banking project

Hot & New

New

Created bySaidhul Shaik

Last updated 7/2026

English

English [Auto],

What you'll learn

Build an end-to-end banking data engineering project on Google Cloud Platform.
Ingest batch data from Cloud SQL to Google Cloud Storage using production-style pipeline design.
Build a streaming ingestion pipeline from Pub/Sub to BigQuery for real-time banking events.
Create Bronze, Silver, and Gold data layers using PySpark on a Dataproc cluster.
Orchestrate banking data pipelines with Apache Airflow DAGs.
Set up CI/CD for Airflow workflows using GitHub and Cloud Build.
Understand how Cloud SQL, GCS, Pub/Sub, BigQuery, Dataproc, PySpark, and Airflow work together in a real project.
Design a portfolio-ready GCP data platform architecture for a banking use case.

Course content

8 sections • 26 lectures • 7h 5m total length

Notes and Project Repository0:05

Project Introduction and Banking Data Platform Architecture30:01
Explore a banking analytics platform on Google Cloud that ingests batch data from Cloud SQL and real-time data from Pub/Sub, processes with Dataflow and Dataproc, and analyzes in BigQuery.
Create a GCP Free Tier Account8:39
Create a Google Cloud Platform free tier account to access $300 in credits for 90 days, complete mobile verification, and enable autopay with a refundable one-time thousand rupees prepayment.
Project Setup and Repository Walkthrough5:58
Understanding the Banking Dataset14:30
Explore the banking dataset on Google Cloud, mapping customer, account, and transaction data from batch and streaming sources, and learn how customers can own multiple accounts and accounts have transactions.

Set Up Source 1 - Cloud SQL MySQL Instance20:52
Set up a Cloud SQL MySQL instance on Google Cloud to host core banking data, create banking database, users, and sample tables: customers, accounts, transactions.
Set Up Source 2 - Pub/Sub18:37
Create and configure Pub/Sub topics and subscriptions, including a dead-letter topic, and generate real-time banking transaction events with a Python producer to ingest into the landing layer.
Set Up the Landing Layer6:10
Set Up the Metadata Layer9:55

Build Ingestion Pipeline 1 - Cloud SQL to GCS (Part 1)11:15
Build Ingestion Pipeline 1 - Cloud SQL to GCS (Part 2)27:49
Design and implement an end-to-end CDC ingestion pipeline from Cloud SQL to GCS using Apache Beam, reading delta changes via watermark, writing Parquet, and updating watermark after success.
Set Up the Bronze Layer (Part 1)11:11
Set up the bronze layer by creating empty BigQuery bronze tables from the GCS landing data and loading with Dataproc PySpark, enforcing schema, deduplication, bronze load timestamps, partitioning, and clustering.
Create a Dataproc Cluster6:52
Load Bronze Data with PySpark on Dataproc31:38

Build Ingestion Pipeline 2 - Pub/Sub to BigQuery49:52
Build ingestion pipeline 2 moves data from Pub/Sub to BigQuery bronze layer using an Apache Beam streaming pipeline on Google Cloud Dataflow, with validation, windowing, deduplication, and dead letter handling.
Set Up the Silver Layer21:19
Set up the silver layer within the medallion architecture by moving bronze data to cleaned silver tables, implementing SCD types 1–3, and merging changes in BigQuery.
Set Up the Gold Layer18:09

Introduction to Airflow Orchestration4:33
Explore how to orchestrate five DAGs in Airflow, scheduling dataflow, beam, spark, and BigQuery SQL jobs from bronze to silver to gold with monitoring.
Create the Banking Ingestion DAG21:30
Learn to build a banking ingestion dag that orchestrates an Apache Beam batch job on Google Cloud Dataflow, moving data from Cloud SQL to GCS and configuring Airflow dags.
Create the Banking Bronze DAG11:23
Create the banking bronze dag that orchestrates a PySpark job on data proc to move data from the landing to the BigQuery branch, starting and stopping the cluster.
Create the Banking Silver DAG7:05
Create the Banking Gold DAG4:12
Orchestrate a gold data pipeline in BigQuery by discovering four SQL files from a GCS path and refreshing silver-to-gold tables: customer 360, daily account balance, daily transaction summary, fraud.
Create the Airflow Environment2:14
Manually Trigger the Airflow DAGs54:35

Requirements

Basic understanding of SQL, Python is helpful.
Basic familiarity with Google Cloud Platform is useful, but the required setup is explained in the course.
No prior real-world data engineering project experience is required. The course is designed to help learners build a complete project step by step.
Learners should have a Google Cloud account to practice the project hands-on.

Description

This course is a hands-on GCP data engineering project built around a realistic banking data platform. Instead of learning services separately, you will see how multiple Google Cloud services work together to solve a complete data engineering use case from source ingestion to curated analytics layers.

You will start with the project architecture and banking dataset, then set up the required GCP resources including Cloud SQL, Pub/Sub, Google Cloud Storage, BigQuery and Dataproc. From there, you will build a batch ingestion pipeline from Cloud SQL to GCS and a streaming ingestion pipeline from Pub/Sub to BigQuery.

The course then moves into medallion-style data processing. You will create Bronze, Silver and Gold layers, load data with PySpark on Dataproc, and prepare business-ready tables that can be used for analytics and reporting. Finally, you will orchestrate the full banking pipeline using Apache Airflow DAGs and learn how CI/CD can be connected with GitHub and Cloud Build.

By the end of the course, you will have a practical, portfolio-ready GCP banking project that you can explain in interviews. This course is especially useful for aspiring data engineers, cloud engineers, ETL developers, data analysts and learners who want real project experience with Google Cloud data engineering tools.

Who this course is for:

Aspiring data engineers who want hands-on project experience with Google Cloud Platform.
Learners preparing for data engineering interviews who need a practical banking project to explain confidently.
Students and freshers who already know basic SQL or Python and want to move into data engineering.
Cloud engineers, data analysts, or ETL developers who want to transition into GCP data engineering roles.
Working professionals who want to learn how batch and streaming pipelines are built on GCP.

A Real Banking Project on Google Cloud for Data Engineers

What you'll learn

Explore related topics

Course content

Course Resources & Materials1 lecture • 1min

Project Introduction and Setup4 lectures • 59min

Source Systems and Landing Layers4 lectures • 56min

Cloud SQL to GCS and Bronze Processing5 lectures • 1hr 29min

Streaming Ingestion and Medallion Layers3 lectures • 1hr 29min

Airflow Orchestration7 lectures • 1hr 46min

CI/CD for the Banking Pipeline1 lecture • 25min

Interview Preparation and Project Storytelling1 lecture • 2min

Requirements

Description

Who this course is for: