Taming Big Data using Spark & Scala

This lecture introduces an additional information section, explains how to access new videos and view the course components, and notes ongoing updates and optimizations within the course ecosystem.

Download the virtual machine image, install the vm software, and configure memory, processors, and a shared folder, with Windows and Mac steps to start, stop, and save state.

Learn how to fix Windows HDFS errors by formatting the namenode from the command prompt, understand data loss risks, and apply two-step remedies for VM storage and active virtual machines.

learn to install and configure MySQL on Windows, set up Hadoop components in a virtual machine, and manage databases, users, and tables while exploring information_schema for data workflows.

Set up IntelliJ for Spark and Scala by downloading the community edition, installing Java and Maven, and adding the Scala plugin for a ready project workspace with dependencies.

Install sbt for your system, create a local Scala project, and run the Scala console and sbt repl to experiment with Spark-based commands.

Explore Hadoop architecture basics, including name node and data node roles, job tracker, secondary name node, and checkpointing, plus data replication, fault tolerance, and commodity hardware benefits.

Learn how to use Sqoop import to transfer data into Hadoop, analyze map-reduce execution, data splitting by min/max values, and monitor job logs and jars for performance.

Learn how to perform Sqoop import in append mode, handling existing targets, choosing between append and overwrite, and validating record counts and metadata during data transfer.

Learn how to use Sqoop import to execute user-specific queries with where clauses, manage range-based splits, and control map counts, including the role of speed by criteria and primary keys.

Discover how sqoop import performs incremental loads using a check column, with append and last modified modes, starting from a first load and then incremental appends using the maximum value.

Master a basic Sqoop import from MySQL to Hive, create a Hive table, and move relational data into Hive using connectivity and import options.

Learn to import data from MySQL to Hive using Sqoop, targeting a specific database, choosing append or overwrite modes, and creating Hive tables with a partition key.

Discover how Hive functions as a data warehouse on top of HDFS, providing a SQL-like Hive query language to execute complex queries on large, file-based data.

Learn how to create and manage Hive partition tables, choose between external and managed partitions, and use static and dynamic partitioning to load data from external sources.

Explore bucketing in hive with external tables and partitions. Learn how bucket counts distribute data across files and how repairs affect reading external data.

Learn Hive performance tuning methods to optimize mapreduce execution, data compression and file formats, partitioning, bucketing, join strategies, and cost-based optimization.

Explain Flume architecture, detailing how clients generate events, agents with source, channel, and sink components perform contextual routing, and how multiple sources, channels, and agents support data aggregation.

Learn Flume configurations, connecting sources to channels and sinks to move events into Hive or other targets. Explore multiplexes and binding sources to channels and sinks in the configuration.

Learn to build a simple Flume job with netcat as the source, a channel and sink, and an agent, streaming data from localhost to the console.

Discover how to build a Flume job using EXEC to execute Unix commands, configure an exec source, and stream file data to a target like a logger, displaying results live.

Learn to run a Flume job on Cloudera that ingests Twitter data, configuring the Flume agent, channel, and sink, with configuration files and console logs.

Learn how flume uses a file channel to prevent data loss, configuring checkpoints and agents, and routing data from Twitter to sinks like logger and hive.

Flume ingests logs and events from web services and ecommerce platforms into Hive for analysis. It emphasizes using Flume to move data without impacting source systems.

Kafka uses topics and partitions spread across brokers to enable parallel writes from producers and reads by consumer groups, with per-partition offsets, leader and follower replication, and zookeeper coordination.

Learn Scala basics via the CLI and REPL on Windows and Cloudera, with mutable and immutable variables, reassignment rules, and debugging tips for Spark code.

Explains object oriented programming in Scala with classes and objects, constructors and instantiation, companion objects, and mutable vs immutable fields with getters and setters.