Learn Big Data Technologies for Complete Beginners

Define big data as a large amount of data not processed by traditional hardware and explore the five v's: volume, variety, velocity, veracity, and value.

Explore Apache Hadoop, an open source framework for distributed storage and processing with HDFS and MapReduce on commodity hardware. Discover components like Spark and Hive.

Identify Hadoop's core components: HDFS for distributed storage, MapReduce for parallel processing, and Yarn for resource management, plus Hadoop common utilities and default block replication for fault tolerance.

MapReduce enables distributed computing by loading data, producing key-value pairs in the map phase, and combining, grouping, and sorting them in the reduce phase.

Learn how map and reduce process data into key-value pairs, shuffle and group them, and build a word count example, plus why Spark offers faster in-memory processing.

Explore the difference between return and yield in Python, learning how return exits a function and yields values over time as a generator, with practical contrasts for MapReduce coding.

Use Google Colab to write MapReduce programs; sign in with your Google account, create a new notebook, and set the title to MapReduce programs while learning how MapReduce works.

Learn how MapReduce counts movies by rating from 1 to 5 using a Python EMR job, with a mapper emitting rating and one, and a reducer summing counts.

Learn how to use MapReduce to compute the sum of order amounts per customer by grouping data by customer ID, mapping lines from a CSV, and reducing with sum.

Apply MapReduce to calculate the average number of friends per age by loading friends.csv, mapping each record by age, and reducing to per-age averages via total divided by count.

Compute the minimum temperature per station using mapreduce by extracting t min values with a mapper and reducing by station id, reading csv input.

Sorts customer spending totals with a two-step mapreduce workflow that swaps keys to sort by order total, using EMR steps and multiple mapper and reducer steps for grouping and sorting.

Implement a MapReduce word count program using EMR, loading and parsing text in the mapper. Reduce by word and sum counts, lowercase words, and split lines by spaces.

Apply regular expressions to extract words and fix word count in MapReduce. Use a \w pattern and the find all method to collect words from each line, ignoring punctuation.

Learn to build a MapReduce word count sorter using multiple mappers and reducers, swapping key and value to sort by count with EMR steps and stopword considerations.

Learn Apache Spark environment setups for cloud and on premise projects, choosing between notebook and Python IDE workflows, with hands-on in Databricks cloud and local development.

Register for the Databricks community edition, verify your email, and set a password to access the free community cloud for hands-on spark learning.

Verify your Databricks account by resetting your password and logging in to the Databricks community edition to explore Spark systems and Spark terminology in the cloud.

Learn to use the Databricks Community Edition workspace to create a Spark notebook in Python, read a CSV with header and inferred schema, and run on a single-node cluster.

Create your first spark application in Databricks by loading a csv with spark.read, using header and infer schema, and displaying data in a dataframe.

Learn Apache Spark by loading a data set, creating a Spark DataFrame, and solving questions with the Spark DataFrame API and Spark SQL.

Learn how databases load data into tables stored as dot dbf files and expose SQL queries, using a three-layer architecture—storage, metadata, and logical layers—for parsing and executing a physical plan.

Explore the differences between Spark table and Spark data frame, including schema on read, persistence, and file formats, and compare sql versus api approaches in distributed storage.

Learn to create and load a spark data frame from a CSV in the Databricks community cloud using a Spark session and a DataFrameReader with header and infer schema.

Learn to create a spark database and table, switch to SQL, query with Spark SQL, and load data from a data frame or parquet.

Convert a data frame into a global temp view and run SQL on the resulting spark table to insert data, highlighting why SQL works on tables, not data frames.

Copy data from a global temporary view into the demo table using insert into, then verify the table now contains the view's data for later sql queries.

Learn Spark SQL by solving ten questions on a demo table, counting and listing distinct fire department call types using SQL with a casted table for faster queries.

Learn spark sql with demos on a demo table: filter delays over five minutes, identify common call types, and compute sums, averages, minimums, and maximums of delay by zip code.

Discover Spark SQL through hands-on examples that extract year and week from call dates, count distinct years, and analyze 2018 neighborhood delay using Spark table and the Spark framework.

Explore Spark data frames and the DataFrame API to reproduce SQL questions, load data with Spark read, and understand transformations, actions, and immutability in a Databricks notebook.

Learn to apply spark sql to a data frame by creating a temp view, then run sql or use the data frame api, or combine both to reproduce results.

Discover the spark driver–executor architecture, learn how actions initiate jobs and return results, and differentiate transformations from functions in data frame workflows.

Explore how Spark reads a CSV into a distributed data frame, partitioned across multiple executors managed by YARN, with the driver coordinating memory, partitions, and in-memory processing.

Explore Spark data processing concepts, including transformations and actions, immutable data frames, lazy evaluation, and the dag of operations that drives narrow and wide dependencies.

Explore Spark DataFrame concepts by applying the DataFrame API, performing transformations and actions, and running SQL on data frames to reveal distinct fire department call types.

Explore Spark data processing APIs, from RDDs to data frame, data set, and Spark SQL, and see how the Catalyst optimizer delivers optimized execution plans as a DAG.

Compare RDD and Spark SQL by showing a counting example using group by country, map, filter, and reduce, then highlight Spark SQL advantages with Catalyst Optimizer and DAG-based execution.

Discover how the spark sql engine analyzes sql and data frame queries, applies cost-based optimization via the catalyst optimizer, and generates efficient java bytecode from rdd operations.

Explore MongoDB, a NoSQL database with collections and documents, learn to install locally, store, retrieve, update, and delete data, including nested documents and JSON-like structures, with fast queries.

Explore how MongoDB stores data in databases, collections and documents, with BSON documents that resemble JSON, support nested data, and assign a unique ObjectId.

Explore MongoDB Compass to view databases, collections, and documents, and learn to connect to a local MongoDB service, create a bookstore database, and insert or update data.

Switch to the bookstore database in the mongo shell, reference the books collection, and insert documents with insert one or insert many, including title, author, pages, genres, rating.

Fetch documents from a MongoDB collection using find and find one, apply filters, and project specific fields for API responses.

Chain MongoDB queries with find, count, limit, and sort to filter by author, limit results to three, and sort titles in ascending or descending order.

Explore nested documents in MongoDB by embedding a reviews array inside a book document and weigh read performance against separate collections, with insertMany workflows and Compass verification.

Learn how to use MongoDB comparison operators to build queries with less than, greater than, and or conditions, filtering by rating and author.

Learn to filter documents with the in operator by matching ratings in a value array, such as 7, 8, or 9, and compare with or and not in.

Learn how to query array fields and nested documents in MongoDB by filtering genres, using array membership and the $all operator, and querying nested reviews with dot notation.

Export a collection, then delete documents with delete one and delete many using the _id or filters, and re-import the saved JSON data to restore the collection.

Update documents in MongoDB using update one and update many, by _id, with $set to modify fields. Use inc, push, pull, and each to adjust values and arrays.

Discover how indexing in MongoDB speeds queries by scanning the index instead of the whole collection, and learn to create, explain, and drop rating-based indexes.