Big Data Hadoop and Spark with Scala

Explore the shift from relational databases to NoSQL, highlighting distributed computing, schema-free design, and fast, scalable handling of unstructured and semi-structured data with DynamoDB, Cassandra, and MongoDB.

Explore the CAP theorem in practice with HBase on Hadoop, comparing consistency, availability, and partition tolerance, and learn how HBase’s column-family NoSQL model handles distributed data with get and put.

Explore how HBase stores data in column families with versions and timestamps, enabling scalable, fast writes and reads on a distributed column oriented architecture with region servers and memstore.

Learn to create and map an HBase table with edge-based storage, load 25,000 transaction records from DFS/HDFS into HBase and Hive, and query with Impala for fast analytics.

Learn to perform a Sqoop import from a MySQL orders table into Hadoop, configuring host, retailer database, and Cloudera user. Inspect the schema with describe and note four mapper partitions.

Learn to import data from MySQL into HDFS with sqoop, perform incremental imports, export back to MySQL, and manage sqoop options and jobs for scalable data movement.

Learn how Flume ingests streaming web server logs and social data into Hadoop, using sources, channels, and sinks, with features like multi-hop, contextual routing, and horizontal scaling.

Configure and run Flume with a source, channel, and sink, using a memory channel and agent to transmit data and log events, including Twitter data workflows.

Learn how Apache Oozie schedules Hadoop jobs with workflow, coordinator, and bundle patterns, using property files to pass arguments and enable load balancing, failover, and callbacks.

Explore Scala, a JVM-based, general-purpose language that combines object-oriented and functional programming with static typing, including mutable and immutable concepts, and its role in Spark and big data.

Learn Spark’s in-memory, scalable analytics from its 2009 Berkeley origins, with Scala focus and Spark Streaming, MLlib, GraphX, and Apache Spark, and compare it with Hadoop MapReduce.

Explore spark practicals in Scala by starting a spark shell, running in local and yarn modes, creating and caching RDDs, and executing actions like collect to view partitions and jobs.

Practice with Spark to load data into an rdd, perform map and flatMap transformations, count words, and collect results for a word count example.

Explore Spark practicals in Scala: load and parse data into place, category, and temperature, and compute per-place minimum temperatures using reduceByKey, with notes on Python equivalents.

Define Spark SQL schemas via reflection or explicit class, then load Yahoo stocks from CSV, clean headers, convert types, and register a dataframe as a temp table for SQL queries.

Learn to analyze flight delay data with spark sql: load csv data, define and apply user defined functions, compute delay percentages, and explore hive queries on big data.

Practice hands-on with Kafka: download, install, rename, and start Kafka; create topics and run a producer and a consumer. Learn how Zookeeper and brokers coordinate messaging and topic delivery.

Explore spark streaming fundamentals, micro-batching, and DStream for real-time data processing, with practical examples and end-to-end integration with Flume and Kafka.

Gain career guidance in data engineering with Hadoop, Spark, Scala, Kafka, and Java and Python options, plus optional Azure Data Factory tools, a personalized project, and resume and interview support.