Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
The Complete GCP Data Engineering Project - Retailer Domain
Highest Rated
Rating: 4.7 out of 5(213 ratings)
1,660 students

The Complete GCP Data Engineering Project - Retailer Domain

Industry Standard Project in Retailer Domain using GCP services like GCS, BigQuery, Dataproc, Composer, GitHub, CICD
Highest Rated
Created bySaidhul Shaik
Last updated 7/2026
English

What you'll learn

  • Understand the End to End Data Engineering Project for Retailer Domain
  • Design and Implement Scalable ETL Pipelines for Healthcare Data
  • Implement Key Techniques like Incremental Data, SCD2, Metadata driven approach, Medallion Arch, Error Handling, CDM , CICD & Many more..
  • Develop and Deploy Data Solutions with CI/CD Practices

Course content

2 sections • 14 lectures • 6h 12m total length

  • Important Links0:02
  • Project Introduction5:07
  • Understanding Project and Direction32:33
  • Lecture 2: Setting up the Data sources – SQL DBs, GCS, BQ, Configs43:13
  • Lecture 3 : Configuring Google Cloud Storage (GCS) as a landing zone9:12
  • Lecture 4: Data Ingestion - Dataproc, Pyspark, GCS Landing-Session11:27:38
  • Lecture 5: Data Ingestion - Dataproc, Pyspark, GCS Landing-Session235:12
  • Lecture 6: Data Ingestion - Dataproc, Pyspark, GCS Landing-Session315:43
  • Lecture 7: BigQuery Bronze Layer11:30
  • Lecture 8: BigQuery Silver Layer25:27
  • Lecture 9: BigQuery Gold Layer13:46
  • Lecture 10: Setting up Airflow DAGS for workflow orchestration27:54
  • Lecture 11: complete CICD with Github, cloud build and airflow1:03:14

Requirements

  • Basic Knowledge on Python and SQL

Description

  • This project focuses on building a data lake in Google Cloud Platform (GCP) for Retailer Domain

  • The goal is to centralize, clean, and transform data from multiple sources, enabling Retailers providers and insurance companies to streamline billing, claims processing, and revenue tracking.

  • GCP Services Used:

    • Google Cloud Storage (GCS): Stores raw and processed data files.

    • BigQuery: Serves as the analytical engine for storing and querying structured data.

    • Dataproc: Used for large-scale data processing with Apache Spark.

    • Cloud Composer (Apache Airflow): Automates ETL pipelines and workflow orchestration.

    • Cloud SQL (MySQL): Stores transactional Electronic Medical Records (EMR) data.

    • GitHub & Cloud Build: Enables version control and CI/CD implementation.

    • CICD (Continuous Integration & Continuous Deployment): Automates deployment pipelines for data processing and ETL workflows.

  • Techniques involved :

    • Metadata Driven Approach

    • SCD type 2 implementation

    • CDM(Common Data Model)

    • Medallion Architecture

    • Logging and Monitoring

    • Error Handling

    • Optimizations

    • CICD implementation

    • many more best practices

  • Data Sources

    • MySQL Retailer Database

    • MySQL Supplier Database

    • API Reviews (api-reviews)


  • Expected Outcomes

    • Efficient Data Pipeline: Automating the ingestion and transformation of RCM data.

    • Structured Data Warehouse: gold tables in BigQuery for analytical queries.

    • After Analysis, Looker BI is used to generate dashboards and reports based on gold-layer tables.

    • All processes (data extraction, loading into GCS, transformation in BigQuery) are managed using Apache Airflow, ensuring automation, scheduling, and monitoring.


Who this course is for:

  • Aspiring Data Engineers, Data Professionals
  • For getting interview Ready

Report abuse