Conquer Big Data with The Ultimate Hadoop & Spark Masterclas

Discover Hive, a data warehousing layer on top of Hadoop, that enables ad hoc querying with HiveQL, external tables, and performance tuning through partitions and bucketing.

Explore Apache Hive on Hadoop, a data warehouse with an sql-like interface, metastore details, and database commands such as show, create, use, and drop.

Demonstrates connecting to Hortonworks via IP and port, authenticating, and using Beeline with Apache Hive to manage databases and tables through create, drop, use, and show commands.

Explore how hive stores data as homogeneous tables with the same schema, and distinguish internal (managed) and external tables, including the metastore and the default location /usr/hive/warehouse on HDFS.

Start hive, create an internal hive table called customer with customer id, cust name, and city as strings, and load data from local or HDFS into the warehouse.

Create an external Hive table that reads all files in an external location; no load data is needed, and dropping the table leaves underlying data files intact.

Learn how the case statement in Hive handles conditional decisions, returning a value from the first true condition, evaluating conditions in order, and using an else (or null) fallback.

Explore partitioning and bucketing in hive to optimize big data queries, dividing tables into date, city, or country slices for faster responses, while bucketing limits partitions to number of buckets.

Explore hive partitioning with a hands-on demo that creates an employee table and partitions data by profession. Learn to load data, enable dynamic partitioning, and query partitions for faster access.

Explore hive built-in functions through practical examples, including round, concat, substr, upper, abs, date diff, date add, date sub, month, and length.

Boost Hive query performance by enabling Tez as execution engine, turning on vectorization and CBO, and selecting efficient file formats like ORC. Cache small tables locally to speed up joins.

Learn to use Sqoop to import data from a MySQL table into HDFS, launching a MapReduce job that transfers two records into the cust input folder.

Learn how to export data from HDFS to an RDBMS using Sqoop export, including configuring connection, user credentials, and --export-dir to move files, then verify results in a MySQL table.

This demo shows importing data from a MySQL staff table into a Hive table via Sqoop, creating a Hive database and storing the staff details on HDFS.

Apache HBase, a distributed column-oriented database on HDFS, scales horizontally to billions of rows with real-time read and write access and a flexible schema, unlike rdbms.

Gain hands-on experience with hbase platform introduction and essential shell operations. Install hadoop and hbase, launch the base shell, and use list, status, version, and create a table.

Master how to create an HBase table with column families, view data with scan, and delete a table through disable and drop, using a sample star table.

Learn to insert data into an HBase table using the put command: create a customer table with personal and professional column families, add rows and columns, and verify with scans.

Demonstrates how to create a hive table and a base table, load local data from employee.txt into a temporary table, and insert overwrite into the base table.

Demonstrates reading data from an HBase base table via a Hive external table, using EMP details and its details column family to query values.

Run a Sqoop import from a MySQL staff table into HBase to transfer data. View results by listing and scanning the staff table, confirming Rajib and Samir are loaded.

Explore how Apache Spark delivers in-memory data processing on Hadoop clusters. Discover the Spark ecosystem—Spark SQL, streaming, Mllib, and graphs—and how Spark uses RDDs and Dag.

Explore how Hadoop and Spark function as distributed processing frameworks for large-scale data, highlighting Spark's in-memory processing and Hadoop's disk-based batch processing with MapReduce.

Explore spark on Databricks in unified data analytics platform to build data pipelines across storage systems and prepare datasets for data scientists and engineers, using community edition to test.

Learn how to install Spark on Ubuntu as part of the Hadoop and Spark masterclass, follow the resources section for setup, and begin working after the setup screen appears.

Learn Spark RDDs as resilient distributed datasets, with lazy transformations like map, flatMap, and filter, and actions such as reduce and collect, plus key operations like groupBy, reduceByKey, and join.

Utilize Spark SQL for in-memory, structured data processing with data frames and SQL across HDFS and Hive, using a scalable distributed engine.

Learn to read a text file with Spark SQL, map to a customer class, create a data frame, and query by selecting name and city while filtering by customer id.

Learn to read and write files in Scala through a simple demo that reads a history file and writes results to a text file using a print writer.

Learn to partition large datasets into partitions based on a condition, then read from a specific partition and identify which values belong to each partition.

Read a json file with Spark SQL, map the path, and create a temporary table named employees for querying the names with a select and basic filters.

Introduce Scala as a general purpose, multi-paradigm language that blends functional and object-oriented programming, compiles to Java bytecode, and runs on the Java virtual machine.

Explore Scala basics, interoperable with Java, and build your program, using data types double, long, short, byte, boolean, and operators including automatic, relational, logical, bitwise, and assignment in IntelliJ project.

Explore if-else constructs, conditional execution, and variable declaration with assignments, using true/false tests and equality checks (x == 20) to drive code paths.

Master the while loop by evaluating a boolean condition and repeating the loop body while the condition is true, demonstrated with a counter incrementing from 10 up to 19.

Describe the do while loop, which executes at least once before checking the condition, and compare it to a while loop that repeats only while the condition is true.

Explore how for loops and for comprehensions act as a control structure to iterate a fixed number of times, using a NUM variable in a 1–10 range.

Define a function as a group of statements that perform a task in escala, create it, accept an integer input from a barometer, and print the value.

Explore anonymous functions and how to assign them to variables, call them in main, and observe how parameter changes alter function values at runtime.

Learn how Scala, a pure object oriented language, uses a company class with name and strength fields, creates an object, and calls a display method to show values.

Explore how Databricks provides a unified data analytics platform, test the community edition to run Spark clusters, and deploy Azure Databricks with autoscaling for testing and learning.

Learn how PySpark provides Python bindings to Spark's distributed processing power, and use Python fundamentals, NumPy and pandas for data tasks, with deeper concepts for machine learning and streaming.

Explore Python as an interpreted, high-level language used in machine learning and big data. Open source with libraries like NumPy, Pandas, Matplotlib, SciPy, scikit-learn, Keras, TensorFlow, and PyTorch.

Choose to use Jupyter Notebook for Python development. Learn how to download, install, and launch Jupyter Notebook in a browser to start working.

Explore how Python uses dynamic and strong typing to treat variables as objects. Learn about integers, floating point numbers, complex numbers, strings, and local versus global variables.

Explains Python operators including arithmetic, comparison, assignment, logical, bitwise, membership, and identity, and shows how to define and call functions with def.

Explore Python's four collection data types—list, tuple, dictionary, and set—and examine their mutability, indexing, key-value access, and unique elements through practical examples.

Learn how Python manages errors by using try, except, and finally blocks to handle built-in exceptions and prevent crashes, with practical examples of raising and catching exceptions.

Explore Python loops, including for and while loops, with nested structures, break and continue, and if statements using logical conditions, and note that Python lacks do-while loops.

Discover essential Python libraries for data science, including NumPy, SciPy, Pandas, Matplotlib, Seaborn, Plotly, scikit-learn, TensorFlow, and more for data manipulation, visualization, and modeling.

Explore object oriented programming concepts—encapsulation, abstraction, polymorphism, and inheritance—and how Python uses classes, the init method, self, and def to implement reusable software.

Read CSV data in Python with Pandas by importing Pandas as pd and loading CSV files into a tabular data frame for easy analysis and conversion to NumPy arrays.

Discover PySpark, the Python API atop Apache Spark, for scalable data processing with RDDs, DataFrames, Spark SQL, and MLlib.

Discover how RDDs underpin PySpark data processing, create an RDD in Databricks, filter names starting with r, and count results.

Explore spark data frames, a distributed table-like structure with rows and named columns, enabling type-safe processing, multi-node scale, and a rich API for filtering, joining, and aggregation.

Explore Spark DataFrames in practice by creating two dataframes, student and marks, with schemas, then joining them, filtering columns, computing average marks, and reading data from a file.