GCP Data Engineering-End to End Project-Healthcare Domain

In this lecture, we introduce the real-world data engineering project. You’ll gain an understanding of the business problem, dataset, and the end-to-end architecture. We will cover:

• Project scope and objectives

• Architecture overview (Data flow from source to consumption)

• Tools & technologies used: GCS, BigQuery, Dataproc, Airflow, Cloud Build

• Expected outcomes and key learning points

Description:
Before we begin processing data, we must configure our data sources. This lecture focuses on:

• Setting up a SQL database (MySQL) as a transactional data source

• Configuring Google Cloud Storage (GCS) as a landing zone

• Creating BigQuery datasets and tables for structured storage

• Setting up configuration files for dynamic pipeline management

This lecture covers the ingestion of raw data into GCS using PySpark on Dataproc. Key topics include:

• Creating and configuring a Dataproc cluster

• Writing PySpark scripts for data extraction and transformation

• Storing ingested data in GCS (Landing Zone)

• Handling errors and logging

Once data is landed in GCS, we need to structure and move it to the Bronze Layer. This session covers:

• Creating partitioned and clustered tables in BigQuery

• Using PySpark/SQL to load raw data from GCS to BigQuery

• Applying basic data validation and schema enforcement

• Automating ingestion with workflow scheduling

The Silver Layer involves refining the raw data from the Bronze Layer by cleaning, deduplicating, and standardizing it. You’ll learn:

• Transforming raw ingested data using BigQuery SQL

• Handling schema evolution and deduplication

• Ensuring data quality with checks and validation

• Optimizing storage for analytical queries

The Gold Layer is where data is modeled for business intelligence and reporting. This lecture focuses on:

• Creating fact and dimension tables for analytical queries

• Applying aggregations and joins for meaningful insights

• Performance tuning and indexing strategies

• Best practices for creating Gold tables for reporting and dashboards

Now that we have an ETL pipeline, we need to automate it using Airflow. This session includes:

• Setting up Cloud Composer (Managed Airflow)

• Writing DAGs (Directed Acyclic Graphs) for workflow orchestration

• Configuring dependencies, retries, and error handling

• Scheduling and monitoring DAG execution

For a production-grade pipeline, we need Continuous Integration and Deployment (CI/CD). This session teaches:

• Version control and branching strategies with GitHub

• Setting up Cloud Build for automated testing and deployment

• Integrating Airflow DAGs with CI/CD pipelines

• Best practices for deployment in GCP