Spark Project on Cloudera Hadoop(CDH) and GCP for Beginners

Discover what big data means through its three defining aspects: data size, velocity, and variety, and see how these drive processing in Hadoop and cloud-based environments.

Explore how Apache Hadoop enables big data processing across a cluster with a simple programming model, distributing work across multiple servers and providing scalable storage and computing.

explains how HDFS stores data as blocks across a cluster, replicates blocks for fault tolerance, and uses the MapReduce programming model to process large datasets.

Explore how Apache Spark delivers a fast, memory-centric execution engine for large-scale data processing across clusters, enabling in-memory partitioned analytics and faster benchmarks such as logistic regression.

Learn spark architecture: how a spark application creates jobs and stages on a cluster with a driver and executors, using scala and python and notebooks to process data with dependencies.

Set up a Cloudera CDH cluster on Google Cloud Platform, install ingestion components and the distributed messaging system, and deploy MongoDB and Cassandra with a local IDE for development.

Create a Google Cloud Platform free trial, set up a Gmail account, verify by phone, and configure billing to launch a virtual machine.

Learn how to create a Google Compute Engine VM, choose region and machine type, install a Linux OS, and configure a static external IP with SSH keys and firewall rules.

Set up a single-node Cloudera Hadoop CDH 6.3 cluster in GCP by installing Java and MySQL, configuring Cloudera Manager, and launching the cluster.

Install Apache NiFi on a single-node CDH 6.3 cluster, download the binary, ensure Java is installed, start NiFi, and access the UI via the cluster endpoint.

Install and configure Apache Kafka on a single node CDH 6.3 cluster, set up ZooKeeper and broker, create a topic, and run producer and consumer to verify messaging.

install apache cassandra on a single node cdh 6.3 cluster on google cloud platform, download cassandra 3.11, configure repository, and create a custom service file to start cassandra.

install MongoDB on a single node CDH 6.3 cluster, enable authentication, start the mongod service, and verify databases and collections for future Python and Spark work.

Install and configure PyCharm Community Edition for PySpark development, set up Spark and Hadoop libraries, create a Python project, and run a simple Spark application.

Install IntelliJ IDEA community edition on Windows, set up a Spark with Scala project, add dependencies, and run a local Spark job from a main Scala object.

Discover resilient distributed datasets (RDD) transformation operations in Spark, including map transformations on lists, using Scala and Python, and learn when to apply action operations like collect.

Explore RDD action operations in Spark, including first, take, reduce, and collect, with practical word-count and filter examples to show how these actions execute on distributed data.

Explore Spark DataFrame operations, from reading data and defining schemas to selecting, filtering, and aggregating with groupBy. Recognize the catalyst optimizer and run on a cluster with spark-submit.

Explore Spark SQL concepts with hands-on exercises that transform data into frames and tables, run queries, and build a simple Facebook program using notebooks.

Explore Apache NiFi to build scalable data pipelines that move data from diverse sources to destinations, transforming and transmitting data in a reliable framework.

Explore Apache NiFi core terminologies, including flow files, processors, process groups, connections, and ports, and learn how to build configurable data pipelines.

Explore Apache NiFi concepts with hands-on practice, building a simple data flow from source to target by dragging processors, configuring connections and templates, and validating a streaming pipeline.

Learn Apache NiFi concepts with hands-on steps to transform a json file, adding a default country field via jolt transformation and specifications.

Explore Apache Kafka, a distributed messaging system that decouples producers and consumers, stores messages for real time processing with offsets, and scales to thousands of messages per second.

Explore key concepts in Apache Kafka: producers send messages to topics, consumers pull from topics, and partitions with offsets and consumer groups enable scalable, non-duplicate processing.

Explore Kafka architecture in multi-node clusters, with producers publishing to topics, partitions, and consumer groups processing streams in real time, plus connectors for databases and file systems integration.

Publish and consume messages with a Kafka producer in Python, creating a topic, configuring a broker, and running a consumer to receive real-time transaction data.

Build a Kafka Python producer and consumer to publish and consume messages on a topic, using a consumer group, and print message keys and values in Google Cloud Platform.

Explore Apache Hive as a data processing layer on Hadoop; learn schema design, partitioning, and joining data with a query language for batch and online analytics.

Build a four-layer data processing pipeline using Spark and Kafka, with Cassandra storage and Dash visualization to simulate real-time retail data and analytics.

Simulate real-time e-commerce retail data using Apache NiFi and a data pipeline, ingest random user and transaction details, transform with Jolt, and publish to Kafka for downstream processing and visualization.

Explore how to consume Apache Kafka messages in real time with Spark Structured Streaming, using Scala, from a broker topic to a console sink for live processing.

Build a data processing pipeline with Spark structured streaming to ingest broker data, deserialize bitstreams into a schema, flatten fields, and write to Cassandra.

Transform streams with Spark structured streaming and write aggregated data frames to MongoDB collections, handling batch-based writes for country sales data and creating multiple collections as needed.

Build a real-time data visualization dashboard with Python and Dash, connecting to MongoDB to stream sales data into interactive bar charts and tables.

Watch an end-to-end project demo showing a real-time data pipeline with Spark, MongoDB, and Kafka, deploying dashboards, building and running a Spark job, and monitoring streaming data.

Install Apache Zeppelin on a CDH 6.3 cluster and use the web-based notebook to run notebooks, connect to Spark and RDBMS databases, and leverage Python and other interpreters.

Learn to build a data analysis pipeline in Apache Zeppelin using Spark SQL, ingest raw transactional data, flatten it, and perform grouped aggregations and visualizations to reveal card-type insights.

Explore Docker containerization by understanding the Docker engine, Docker images, and containers, and compare container performance and isolation with traditional virtual machines.

Install Docker on Ubuntu 18.04 by updating the system, setting up the Docker repository, installing the Docker Engine and daemon, verifying the installation, and running a container from Docker Hub.

Install docker desktop on Windows, ensure a 64-bit processor with virtualization enabled, then log in to download and set up; use Docker Toolbox on Windows Home edition as a workaround.

Learn practical docker usage by pulling images, running and managing containers, building custom images with Dockerfile, and working with a MySQL container to create databases.