Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Apache Kafka - Spark - Cassandra Real Time Streaming Project
Rating: 4.2 out of 5(35 ratings)
1,105 students

Apache Kafka - Spark - Cassandra Real Time Streaming Project

Real Time Data Streaming Project, that involved building the architecture from scratch and deployment.
Created bySonal Saxena
Last updated 3/2023
English
English [Auto],

What you'll learn

  • Professionals looking for an end to end kafka, spark and cassandra streaming pipeline.
  • Anyone who wants to understand how to use Apache kafka in their current architecture.
  • Engineers who are looking to design a Kafka Solution Pipeline.
  • A complete case study from start to execution.

Course content

3 sections • 14 lectures • 32m total length

  • Introduction: Project_Overview0:52

    Explore building a real-time streaming pipeline using Kafka for data streaming, Spark plus Scala for processing, and Cassandra for storage, with Ubuntu setup and end-to-end execution.

  • Documents0:38

    Consult the course documents to access the command list used throughout the course, available in the resources accompanying the video lecture and to assist during software installations and pipeline creation.

  • Ubuntu_Machine_Creation1:53

    Rent a ubuntu machine on digital ocean, sign up, create a project, and set up a droplet with a password. Launch the ubuntu machine when ready and terminate when done.

  • Putty_Installation1:14

    Download and install PuTTY, then connect to your Ubuntu machine on DigitalOcean using the root username and the IPv4 address, entering the password to log in.

  • Java_Installation1:45

    Install Java on Ubuntu first, run sudo apt-get update, verify with java -version, then proceed with installing Kafka, Hadoop, Spark, and Cassandra for an end-to-end streaming pipeline.

  • Kafka_Setup4:35

    Install and configure Apache Kafka with Zookeeper to enable real-time streaming pipelines; create directories, download and unzip Kafka, update config with IPv4 address, then start Zookeeper and Kafka to test.

  • Hadoop_Installation3:02

    Install hadoop on Ubuntu by downloading the hadoop tar archive, moving it to /usr/local/hadoop, updating .bashrc with the hadoop bin path, sourcing the changes, and verifying installation.

  • SPARK SCALA Installation2:25

    Install Spark 2.4.8 with Scala 2.11, ensure compatibility with the Cassandra Connector, update the bashrc with Spark path, source changes, and launch Spark shell.

  • Cassandra_Installation2:54

    Install Apache Cassandra, the NoSQL database, on Ubuntu by adding the repository, installing apt-transport-https, importing the GPG key, updating repositories, and enabling the service, then connect to Cassandra via SQL command.

Requirements

  • Baisc understanding of Kafka Architecure, Spark and Cassandra or SQL

Description

The course is designed to provide a comprehensive understanding of real-time big data processing using Kafka, Spark, and Cassandra. In today's world, data is produced at an unprecedented rate, and the ability to process and analyze this data in real-time is critical for making informed decisions. This course focuses on the fundamental concepts and architecture of Kafka, Spark, and Cassandra, and how they work together to create a robust big data processing pipeline.


Students will learn how to set up Kafka clusters and work with Kafka producers and consumers. Students will also learn about Kafka Streams, a client library for building real-time streaming applications that process data directly within Kafka.


Throughout the course, students will gain hands-on experience through practical exercises and projects that simulate real-world scenarios. By the end of the course, students will have a understanding of how to use Kafka, Spark, and Cassandra to build real-time big data processing systems.


Course Objectives:

  • Understand the fundamental concepts of real-time big data processing

  • Learn the architecture setup of Kafka, Spark, and Cassandra

  • Understand how Kafka, Spark, and Cassandra work together to create a real-time big data processing pipeline

  • Gain hands-on experience with Kafka, Spark, and Cassandra through practical exercises and projects

  • Learn how to build a real-time big data processing pipeline from scratch


This course is intended for software engineers, data engineers, and data analysts who have a basic understanding of programming concepts and are familiar with SQL.

Who this course is for:

  • Engineers looking for a CASE STUDY on Real Time Data Streaming involving KAKFA and SPARK

Report abuse