Taming Big Data using Spark & Scala

Explore the different forms of big data platforms and practice setups for Spark and Scala, including local Windows installations, cloud sandboxes, and Hadoop.

Learn how to install and set up big data tools on Windows or Cloudera using complete working virtual machines, with downloadable options and guided setup.

Explore the shared course content for taming big data with Spark and Scala: review folder structures, Scala dependencies, and how to run Spark jobs on cloud or Windows virtual machines.

This lecture introduces an additional information section, explains how to access new videos and view the course components, and notes ongoing updates and optimizations within the course ecosystem.

Download the virtual machine image, install the vm software, and configure memory, processors, and a shared folder, with Windows and Mac steps to start, stop, and save state.

Install and configure IntelliJ on a Windows VM by downloading the installer from the provided link, completing setup, launching the IDE, and adjusting basic settings to begin development.

Learn how to fix Windows HDFS errors by formatting the namenode from the command prompt, understand data loss risks, and apply two-step remedies for VM storage and active virtual machines.

learn to install and configure MySQL on Windows, set up Hadoop components in a virtual machine, and manage databases, users, and tables while exploring information_schema for data workflows.

Set up IntelliJ for Spark and Scala by downloading the community edition, installing Java and Maven, and adding the Scala plugin for a ready project workspace with dependencies.

Set up IntelliJ for spark and scala by importing dependencies, selecting compatible spark versions from the maven repository, and installing the plugin to enable smooth development.

Identify common IntelliJ setup issues in Spark and Scala projects, including missing plugins and settings problems, and show how to use the source tab and right-click to inspect.

Install sbt for your system, create a local Scala project, and run the Scala console and sbt repl to experiment with Spark-based commands.

Demonstrates setting up winutils on Windows, configuring Java home and environment variables, and configuring Hadoop-like settings so Spark runs against a local filesystem.

Explore Hadoop architecture basics, including name node and data node roles, job tracker, secondary name node, and checkpointing, plus data replication, fault tolerance, and commodity hardware benefits.

Refresh on Hadoop's distributed file system, blocks, and replication, and explore how Hive, MapReduce, and external tables read data stored in text, sequence files, Avro formats.

Explore Hadoop terminologies and configurations, including block size, data blocks, replication, name node concepts, speculative jobs, archiving, and map reduce settings for fault-tolerant, scalable storage.

Learn how to use Sqoop import to transfer data into Hadoop, analyze map-reduce execution, data splitting by min/max values, and monitor job logs and jars for performance.

Learn how to perform Sqoop import in append mode, handling existing targets, choosing between append and overwrite, and validating record counts and metadata during data transfer.

Learn how to use Sqoop import with the overwrite option and delete target directory, and explore how text, Avro, and Parquet formats affect data storage and schema.

Learn how to use Sqoop import to execute user-specific queries with where clauses, manage range-based splits, and control map counts, including the role of speed by criteria and primary keys.

Discover how sqoop import performs incremental loads using a check column, with append and last modified modes, starting from a first load and then incremental appends using the maximum value.

Learn how to create, list, and execute Sqoop jobs, manage group jobs, and run commands to refresh and monitor job execution in a big data environment.

Demonstrate using Sqoop import all to transfer all MySQL tables to HDFS, configuring maps, primary keys, and table exclusions, while managing target directories on first import.

Master a basic Sqoop import from MySQL to Hive, create a Hive table, and move relational data into Hive using connectivity and import options.

Learn to import data from MySQL to Hive using Sqoop, targeting a specific database, choosing append or overwrite modes, and creating Hive tables with a partition key.

Learn to use Sqoop export to move data from HDFS to MySQL, export tables, manage primary keys, and perform incremental loads with update mode or update only.

Discover how Hive functions as a data warehouse on top of HDFS, providing a SQL-like Hive query language to execute complex queries on large, file-based data.

Explore how Hive partitioning and bucketing organize data into partitions and files to speed access, covering static and dynamic partitions and bucketing concepts.

Explore Hive query language, a SQL-like syntax for loading data and performing queries with select, join, and group by. Understand databases, tables, external vs managed, and static and dynamic partitioning.

Practice Hive ql by creating databases and tables, dropping objects, and loading sample data, while exploring managed and external tables and their file system locations.

Learn how to load huge data into Hive QL managed tables, using scoop, creating category tables, and managing data formats with append versus overwrite.

Explore creating and loading managed and external Hive tables, including external data sources, file formats, and the behavior of data on drop, with scoop import.

Learn how to create and manage Hive partition tables, choose between external and managed partitions, and use static and dynamic partitioning to load data from external sources.

Explore bucketing in hive with external tables and partitions. Learn how bucket counts distribute data across files and how repairs affect reading external data.

Learn Hive performance tuning methods to optimize mapreduce execution, data compression and file formats, partitioning, bucketing, join strategies, and cost-based optimization.

Explore Flume as a data ingestion tool for collecting, aggregating, and transporting large data, detailing its architecture, core components, channels, contextual routing, reliability, and logs use cases.

Explain Flume architecture, detailing how clients generate events, agents with source, channel, and sink components perform contextual routing, and how multiple sources, channels, and agents support data aggregation.

Explore Flume data flows, including multi-hop and fan-in/out, and how contextual routing uses interceptors and channel selectors to route data reliably from sources to sinks.

Learn Flume configurations, connecting sources to channels and sinks to move events into Hive or other targets. Explore multiplexes and binding sources to channels and sinks in the configuration.

Learn to build a simple Flume job with netcat as the source, a channel and sink, and an agent, streaming data from localhost to the console.

Discover how to build a Flume job using EXEC to execute Unix commands, configure an exec source, and stream file data to a target like a logger, displaying results live.

Learn to configure a Flume job with a sequence generator, first displaying the generated numbers on screen, then generating data and writing it to DFS in two phases.

Deploy a Flume job using a sequence generator source that writes to HDFS via the sink, configure the path correctly, and observe the generated files and their partitions.

Configure a Flume job on Windows to ingest Twitter data into HDFS using a Twitter source, memory channel, and DFS sink; set up a Twitter app and keys, then run.

Learn to run a Flume job on Cloudera that ingests Twitter data, configuring the Flume agent, channel, and sink, with configuration files and console logs.

Learn how flume uses a file channel to prevent data loss, configuring checkpoints and agents, and routing data from Twitter to sinks like logger and hive.

Explore practical Flume setups to ingest Twitter data into a hive sink, configure sources and codecs, and manage partitioning to load streams into hive tables.

Flume ingests logs and events from web services and ecommerce platforms into Hive for analysis. It emphasizes using Flume to move data without impacting source systems.

Kafka uses topics and partitions spread across brokers to enable parallel writes from producers and reads by consumer groups, with per-partition offsets, leader and follower replication, and zookeeper coordination.

Learn Scala basics via the CLI and REPL on Windows and Cloudera, with mutable and immutable variables, reassignment rules, and debugging tips for Spark code.

Explore mutable vs immutable variables in Scala, create and type variables, and apply string operations like equals, toLowerCase, reverse, substring, and split, with examples for data frame and database contexts.

Explore Scala ranges as immutable collections, using inclusive and exclusive forms, convert ranges to a list, array, or dataset, and apply for each element and max operations.

Explore Scala functions, including named, anonymous, and one-line definitions, with examples of sum of squares and partial or lambda expressions.

Explains object oriented programming in Scala with classes and objects, constructors and instantiation, companion objects, and mutable vs immutable fields with getters and setters.