
Explore data ingestion to Google Cloud Platform using the console, Cloud SDK, and client libraries; learn to create a project, set up a bucket, upload files, and access gsutil URLs.
Explore Google Cloud client libraries, with a Python focus; learn to enable storage API, install the Python library, and upload files using a Docker container and a service account credential.
Generate a 1000-row json data set with id, first name, last name, email, and gender; transform gender to M/F and publish to a GCP bucket via NiFi using a UUID.
Explore an end-to-end data engineering demo that ingests simulated data into cloud storage, triggers cloud functions, transforms gender to F or M, and loads into Cloud SQL.
Explore how to set up and use BigQuery as a fully managed, serverless data warehouse; learn to create datasets and tables, define schemas, and perform basic data transformations.
You are going to learn about how we can create data engineering solutions in Google Cloud Platform (GCP) using different available tools, mainly using Python as our core programming language. We will start from basic introduction of what each component is mainly used for, then dive straight into demos with detailed explanation on the design choice and reasonings behind. I am confident that you will come out of the course with a better understanding on what each component could possibly do and drive data engineering solutions in your team!
There are the list of tech stacks that we will be covering
Integrating with GCP
Google Cloud Platform Console, Cloud SDK and Client Libraries
Data Storage/ Messaging
Google Cloud Storage, Google Pub/Sub
Databases
Cloud SQL, BigQuery, Bigtable
Data Processing
Cloud Functions, Cloud Dataflow, Cloud Dataproc
I have created a few specialised labs to integrate all the tools mentioned above instead of telling you what each component could do! It could help to accelerate your understanding so you can apply on your existing GCP workflow immediately. Detailed code and steps will also be shared in the video as well as self-written articles, so just sit back, relax and enjoy all the demos!
Let's get onboard on the cloud journey now!