Streaming data pipeline using Confluent Kafka & Google Cloud

Hands on course - Real Time Data Streaming Pipeline using Confluent Kafka, GCP & Looker from scratch

Created byNilesh Shukla

Last updated 1/2026

English

English [Auto],

What you'll learn

Able to build Enterprise Grade, scalable data pipeline using Confluent Kafka & Google Cloud Platforms
Configure, use and trouble shoot setting up of Google cloud Platform services like Buckets, Cloud Data Fusion
Configure, use and trouble shoot setting up of Confluent Cluster, Topic, Keys etc
Configure, use and trouble shoot setting up of Confluent managed Sink & Source Connectors
Configure, use and trouble shoot setting up of Google cloud Platform reporting services like Looker

Course content

6 sections • 15 lectures • 2h 1m total length

Introduction & Data PipelineArchitecture Overview5:13
Build an analytics data pipeline from Confluent Kafka to Google Cloud, ingesting JSON data, syncing hourly to Cloud Storage, transforming with Cloud Data Fusion, loading BigQuery, and visualizing in Looker.

Confluent: Create Account, Cluster, & API Key3:44
Create a confluent account, set up a Google Cloud Platform cluster, and generate an API key to connect to the cluster, with guidance on endpoints and key management.
Confluent: Create Topic, Fully Managed Datagen Source Connector5:51
Create a fully managed Datagen source connector in Confluent Kafka, generating sample transactions data in JSON and pushing it into a topic.

GCP: Create Bucket on Cloud Storage on GCP2:13
Create a cloud storage bucket on Google Cloud Platform for the confluent kafka data pipeline, named confluent_gcp_data, with a hierarchical namespace, single region us east one, and public access blocked.
GCP: Create Service Account / Key for Cloud Storage1:53
Create a Google Cloud service account and download a JSON key to enable the Confluent Kafka connector to access a Cloud Storage bucket for the streaming data pipeline.
Confluent: create fully managed Google Cloud storage sink connector8:10
Create a fully managed Google Cloud Storage sink connector to move data from a confluent topic to a GCP bucket, using JSON output and hourly path formatting.

GCP: Create BigQuery Dataset and Table2:36
Create a BigQuery dataset and table to support a streaming data pipeline with Confluent Kafka on Google Cloud, and connect Looker for dynamic operational reports using SQL queries.
GCP: Verify data flow from confluent Kafka topic to GCP storage bucket7:02
Demonstrate validating a Confluent to Google Cloud data pipeline, from source connector to Confluent topic and storage bucket, enabling Cloud Data Fusion, BigQuery, and Looker Studio analytics.
GCP: Create Cloud Data Fusion instance for ETL processing2:56
Learn to create a Cloud Data Fusion instance on Google Cloud to run ETL for a streaming pipeline, including enabling APIs, authorization, and monitoring instance creation.
GCP: Build ETL Pipeline in Cloud Data Fusion - INSERT Mode11:38
Build a streaming data pipeline from Google Cloud Storage to BigQuery using Confluent and Cloud Data Fusion Studio, configure GCS source, JSON schema, and insert mode, then deploy and schedule.
GCP: Build ETL Pipeline in Cloud Data Fusion - UPSERT Mode26:15
Develop and deploy an upsert-enabled ETL pipeline in Google Cloud Data Fusion that ingests JSON from Google Cloud Storage buckets via Confluent Kafka into BigQuery for analytics with Looker Studio.
GCP: Deploy & Execute Data Pipeline19:41
Build and execute a streaming data pipeline using Confluent Kafka and Google Cloud, moving data from Cloud Data Fusion to Google Cloud Storage and into BigQuery for analytics in Looker.

Requirements

No programming experience needed
Able to create free confluent kafka account to follow along
Able to create free GCP - Google Cloud Platform account to follow along

Description

Course Overview

In this hands-on course, participants will follow a step-by-step approach to build a real-time streaming analytics data pipeline that integrates Confluent Kafka with Google Cloud Platform services. Learners will design a pipeline that streams data from Kafka into Google Cloud Storage, processes it with Dataflow, stores it in BigQuery, and finally visualizes operational insights using Looker Studio. This practical journey ensures participants not only grasp the concepts but also apply them directly in a cloud-native environment.

Google Cloud Platform Experience

On the GCP side, participants will gain exposure to key services including Cloud Storage, Dataflow, BigQuery, and Looker Studio. They will also learn essential skills in logging, monitoring, troubleshooting, and configuration, which are critical for building secure, scalable, and reliable streaming applications in the cloud.

Confluent Kafka Experience

On the Kafka side, learners will set up a fully managed Kafka cluster, create topics for message streaming, configure a Datagen Source Connector to simulate real-world data ingestion, and integrate with GCP Storage Sink Connector. This provides valuable hands-on experience with enterprise-grade Kafka infrastructure—without the complexity of managing it manually.

Learning Outcomes

By the end of the course, participants will have:

Built a working, scalable streaming analytics pipeline.
Gained insights into cloud-native architectures and streaming design patterns.
Acquired hands-on skills directly applicable to real-world projects and professional roles.

Who Should Attend

This course is ideal for Cloud Engineers, Data Engineers, Product Owners, Product Managers, Scrum Masters, and Technology Leaders seeking practical experience in building real-time streaming analytics pipelines with Confluent Kafka and Google Cloud.

Who this course is for:

Entry Level Data Engineers who want to learn about building Data pipelines using Confluent Kafka & Google Cloud Platforms
Entry Level Cloud Engineers who want to learn about building Data pipelines using Confluent Kafka & Google Cloud Platforms
Mid to Sr. Level Data Engineers who want to get exposure on building Data pipelines using Confluent Kafka & Google Cloud Platforms
Product Managers/Product Owners who want to get exposure on building Data pipelines using Confluent Kafka & Google Cloud Platforms
Technology Leaders who want to get exposure on building Data pipelines using Confluent Kafka & Google Cloud Platforms

Streaming data pipeline using Confluent Kafka & Google Cloud

What you'll learn

Explore related topics

Course content

Introduction - Business Use Case & Architecture Overview1 lecture • 5min

Confluent Cloud - Kafka Setup2 lectures • 10min

Google Cloud Platform Services & Kafka Integration Setup3 lectures • 12min

Data Pipeline in Action6 lectures • 1hr 10min

Visualization & Operational Reporting2 lectures • 17min

Wrap Up and Next Steps1 lecture • 8min

Requirements

Description

Who this course is for: