Hands-On Data Engineering in Google Cloud Platform | Python

Explore data ingestion to Google Cloud Platform using the console, Cloud SDK, and client libraries; learn to create a project, set up a bucket, upload files, and access gsutil URLs.

Learn to install cloud sdk via docker, authenticate inside the container, and use gcloud, gsutil, and bq to list projects, access cloud storage, and query public datasets.

Explore Google Cloud client libraries, with a Python focus; learn to enable storage API, install the Python library, and upload files using a Docker container and a service account credential.

Learn to install Apache NiFi on a Google Cloud Platform VM using Docker, expose its UI on port 9090, and visualize end-to-end data flows between on-premise and cloud environments.

This NiFi demo connects to Google Cloud via a service account JSON, configures write permissions to Google Cloud Storage and Pub/Sub, and pushes generated files to Google Cloud Storage.

Explore the Google Cloud Platform console’s drag-and-drop upload to cloud storage, the Cloud SDK tools gsutil and gcloud, and Python client libraries with an ETL tool.

Load raw data directly into Google Cloud Platform to enable cross-project sharing and retire on-premise compute, using cloud storage for batch and pub/sub for streaming with security and lifecycle options.

Explore pub/sub, a messaging middleware for publisher–subscriber systems, enabling real-time streaming and IoT use cases with at least-once delivery, up to 2 gb/s throughput, and retention up to seven days.

Explore pub/sub on Google Cloud Platform by creating a topic and a default subscription, publishing messages from a publisher, and consuming them as a subscriber in real time with acknowledgement.

Explore serverless cloud functions in Google Cloud, triggerable by Cloud Storage and other events, to transform data in Python and load results into Cloud SQL in an end-to-end demo.

Generate a 1000-row json data set with id, first name, last name, email, and gender; transform gender to M/F and publish to a GCP bucket via NiFi using a UUID.

Set up a Cloud SQL instance on Google Cloud Platform with MySQL, configure instance details, enable required APIs, create a my_company database and an employee table via Cloud Shell.

Set up cloud functions to connect to cloud sql and download objects from google cloud storage, using python, and deploy with dynamic variables and logging for testing.

Explore an end-to-end data engineering demo that ingests simulated data into cloud storage, triggers cloud functions, transforms gender to F or M, and loads into Cloud SQL.

Explore cloud dataflow and the Apache Beam SDK for building batch and streaming pipelines in Python. Trace input through transformations to output in BigQuery, and explore windowing and triggers.

Set up source data for a data flow by creating a Pub/Sub topic and subscription, and configuring NiFi to generate synthetic files with retry and replace text steps.

Explore how to set up and use BigQuery as a fully managed, serverless data warehouse; learn to create datasets and tables, define schemas, and perform basic data transformations.

Learn to set up Apache Beam development in Google Cloud Platform using notebooks for quick local development, including enabling Dataflow API, using Jupyter notebooks, and testing streaming pipelines with Pub/Sub.

Deploy dataflow jobs from a GCP notebook on Google Cloud Platform by creating a GCS bucket for staging and temp files, configuring a streaming pipeline to BigQuery.

Explore an end-to-end data engineering demo that streams dummy data from Pub/Sub through Dataflow to BigQuery, with monitoring, latency metrics, and practical tips.

Explore how Cloud Dataproc, a managed service for Spark, Flink, Presto, Hadoop and 30+ tools, enables quick cluster spins, scalable workflows, and integration with BigQuery, Cloud Storage, and Bigtable.

Set up source data by downloading the Alice in Wonderland plain text from Gutenberg.org, saving as input.txt, and creating a Google Cloud Platform storage bucket for a Cloud Dataproc demo.

Set up a cloud function to trigger a PySpark script in cloud storage, deploy a dataproc cluster, and output a word count CSV after text cleanup.

Set up a cloud function to trigger a PySpark job on a Dataproc cluster via a cloud storage event, passing variables, and publishing results to Pub/Sub for end-to-end data processing.

Learn to set up Google Cloud Bigtable, a fully managed NoSQL database, including creating an instance and a simple table schema with a row key, column family, and count.

Set up a cloud function with a pub/sub and storage trigger to ingest a csv from cloud storage into Bigtable, parsing rows and bulk inserting word counts.

Explore end-to-end data engineering on Google Cloud Platform using Cloud Storage, Pub/Sub, Cloud Functions, Dataflow, and Dataproc, with NiFi orchestration and databases like BigQuery and Bigtable for real-time and batch pipelines.