50 Hrs Big Data Mastery: PySpark, AWS, Scala & Data Scraping

Explore Scala, a concise high-level language that blends object-oriented and functional programming. Learn its compatibility with the JVM, access to Java libraries, and how Scala integrates these paradigms.

explore basic arithmetic with variables by declaring integers, performing addition, subtraction, multiplication, and division, and printing results to the console, including integer division behavior.

Practice a quick Scala quiz by declaring three integer variables A, B, and C and implementing an arithmetic equation to check your understanding.

Take user input in Scala using the read line function, treat input as a string by default, and convert to integers to sum two numbers, avoiding string concatenation.

Explore flow statements in scala, including if-else AFL statements and loops, which control conditional execution and repeat code blocks, with examples like birthday wishes and continue statements.

Learn flow control with if else statements and how conditions decide which code runs. See examples that compare numbers and print outcomes.

Master flow control with an if statement by building a quiz that checks the playland entrance age, allows entry only if older than 13, and prints welcome or underage messages.

Master nested if statements to control flow with conditions, inputs, and else branches. Practice prompting for two numbers, testing greater than 10, and computing sums when conditions hold.

Demonstrates flow control using logical operators by prompting age and height, applying an and condition to allow entry if age is over 30 and height is at least five feet.

Master flow control with if, else, and else if in Scala, learning how to evaluate conditions and nest checks to guide program execution.

Explore how the break statement stops loops in for and while constructs by evaluating a condition inside the loop, including a zero exit and a running sum.

Explore flow control with a fortune game project, using loops and if statements to provide hints, track five guesses, and handle win, loss, and random number generation.

Design a Scala flow-control project building a number-guessing game with a 0-100 X, using for or while loops and if-else logic, plus random number generation.

This lecture demonstrates flow control in the project, using a for loop and a game status variable to determine win or loss, including future random number generation.

Explore how a function is a reusable block of code with parameters, a return type, and a body, and learn how to declare it in Scala to avoid repetition.

Learn to write a basic function that takes two integers, compares them with an if statement, and returns the greater value, including input handling and function calls.

Learn how named arguments let you pass function parameters in any order by mapping values to parameter names. This technique helps manage functions with many parameters and avoids type mismatch.

Learn to implement Python functions for bill calculation: get bill amount, get discount, apply discount with a default of $10 when zero, and print both discounted and actual bills.

Implement four anonymous two-parameter functions for add, subtract, multiply, and divide, then integrate them into a complete equation and print the result.

Explore scope in programming by showing how variables declared inside braces are accessible within those blocks and how global variables differ from local ones across functions.

Design a simple menu-driven program that prompts the user to check balance, withdraw, deposit, or quit. Start with a main function and refactor into balance, withdraw, and deposit functions.

Explore the use of a global balance variable inside functions to check balance, withdraw, and deposit funds, and update the balance accordingly.

Break the loop when credentials are valid and extract code into modular functions for taking credentials, showing the menu, and making transactions, improving readability and maintainability.

Create classes that host variables and functions, access them via methods, and instantiate objects with constructors. Define function to print data and another to return the name with greater semester.

Practice building a class number that stores a value and exposes comparison methods with another class object. Return true if the parameter’s value is greater than the calling instance.

Learn how Scala lists are immutable and require creating new lists to add elements, using start and end appends, and nesting lists to build complex structures.

Learn to extract elements from a list by slicing, taking the first few elements into a new list up to a given index without changing the original.

Prompt users for item, price, and count, cast types, create objects, and append them to a list buffer, turning input into structured data for processing.

Learn to implement required functions inside a class and manage data with a list buffer to print items, prices, counts, and compute the total grocery bill.

Learn how to add and remove items in a set, preserving only unique elements using the plus-equals and minus-equals notation, with practical examples.

Explore the stack data structure, a last-in, first-out structure where you add and remove elements from the top, with a focus on Scala's mutable stack and its basic syntax.

Explore push and pop operations in stacks, including top of the stack access, as you build and manipulate a stack with integers, printing results to understand data structure behavior.

Develop a mini project to understand stacked data structures by building an equation bracket validator that checks valid opening and closing brackets using a stack.

Validate a bracketed expression with a stack in Scala. Address extra opening or closing brackets using a validation flag and empty stack checks.

Explore the Hadoop ecosystem, core concepts HDFS, YARN, and MapReduce, and how Spark distributes data across multiple machines with flow analysis that speeds up processing compared to MapReduce.

Explore Spark architecture by understanding the driver (master) node, cluster manager, and worker nodes, how tasks and transformations are distributed, executed, and returned as final output.

Explore the spark ecosystem, from Spark SQL for transforming data and querying as tables, to Spark Streaming for real-time inputs, MLlib for machine learning, and GraphX for graph visualization.

Create a Databricks account using the community edition, verify your email, and sign in to explore notebooks and begin writing Spark code.

Set up a Databricks cluster, attach it to a notebook, and write a hello world in Scala to verify the environment.

Set up Spark and Hadoop on Windows by installing required utilities, configuring Hadoop home, Spark home, and Java home, and launching the Spark shell to verify a clean environment.

Create a Spark session and read a CSV into a data frame with header true, then compare dataframe reading with Spark context.

Demonstrates writing a Spark DataFrame to a file or folder with df.write, setting header and format options, using overwrite mode, and reading back from file or folder with Spark.

Explore Hadoop and Spark ecosystems, detailing Spark architecture with driver and cluster managers, and how workers execute tasks across languages and libraries like Spark SQL, Spark Streaming, MLlib, and GraphX.

Validate spark installation on Windows by launching Spark Shell and PySpark, verify Spark version 3.1.x, and prep for writing PySpark code, with Databricks workflow upcoming.

Install the JDK on macOS using the installation wizard, click continue, enter your password, and confirm a successful installation.

Install Python on Mac to continue setting up the big data environment; download Python 3.9.6 from the Mac download page and run the installer.

Master Spark RDD map by applying a lambda to each element to produce a new RDD, with examples like splitting strings by spaces and appending text.

Discover how to replace a lambda with a regular function in Spark RDDs map, split strings, convert to integers, and build robust map workflows.

Learn how to replicate an RDD map operation using a lambda function in Spark, building a split and length-based transformation with list comprehension for readable, concise code.

Explore the groupByKey transformation on Spark RDDs by converting data to key–value notation, using map or flatMap to create (key, value) pairs, and collecting values into grouped lists per key.

Learn how to manage Spark RDD partitions by repartitioning and coalescing, understand when to increase or decrease partitions, and see how partitioning affects read and write performance.

Compute the minimum and maximum ratings per movie using Spark RDDs with map and reduceByKey, converting strings to key-value pairs and applying a lambda for min and max.

Learn to filter Spark DataFrames by rows using filter and where, apply single and multiple conditions, and use is in, starts with, ends with, contains, and like with column expressions.

Explore Spark df sorting with sort and orderBy, applying ascending or descending orders on single or multiple columns. Understand interchangeable notations and the integer data requirement for accurate sorting.

Read file into Spark dataframe, then use group by to display counts, male and female splits, marks by gender, and min, max, and average marks by course and age group.

Learn how Spark dataframes use caching and persist to store intermediate results in memory. See how actions trigger evaluation and subsequent transformations read from the cache, speeding up workflows.

Learn to refer to the underlying rdd instead of the dataframe and perform operations on it, including converting between dataframe and rdd and grouping by multiple columns.

Learn how to write a Spark DataFrame back to memory or an output directory, control through write options, modes (overwrite, append, ignore, error), and read data back.

Spark DFs project reads a file into a dataframe, counts employees, derives unique departments using group by or select with dropDuplicates, and prints department names.

Explore a collaborative filtering dataset by loading movie ratings into Databricks, configuring Spark read options, and inspecting the resulting data frame to start collaborative filtering workflows.

Split the ratings dataframe into training and test sets using an 80/20 random split to train a collaborative filtering model and evaluate the recommender system.

Explore collaborative filtering with hyperparameter tuning and cross-validation by building multiple models, evaluating with root mean squared error, and using a grid builder and cross validator to find parameters.

Explore Spark Streaming df aggregations by performing a group by and count on a dataframe, observing how new files update the word counts in real time in Databricks.

Explore the etl pipeline with spark as the driver that extracts data from diverse sources, optionally transforms it, and loads it to a chosen output format or destination.

This video demonstrates extracting data in the ETL pipeline by reading a text file into a data frame, displaying the results, and outlining a subsequent word count transformation before loading.

Transform data in the ETL pipeline by converting lines into word lists, exploding them into individual words, and counting occurrences to produce a word frequency result.

Introduce change data capture (cdc) and a pipeline to capture and replicate all changes from a database into storage, outlining the architecture for the end-to-end cdc project.

Execute change data capture by creating and testing a data migration service source endpoint for a MySQL database, then name and verify the endpoint before configuring the destination.

Create a DMS replication instance and an end point to enable change data capture and data migration, using the minimal available instance size in the VPC for the DMF task.

Establish a MySQL Workbench connection to the target database, create the schema and a primary key table, and run the dump to enable change data capture and ongoing replication.

Stop ongoing change data capture replication and related instances to prevent costs, then create a spark job to read data from the three and write to the three buckets.

Create a glue job in Databricks to perform a full load and updates for change data capture, reading full and updated data, renaming columns, and writing final output with overwrite.

extract the bucket name and file name from the lambda event to identify which S3 object triggers the function, and verify by uploading a file.

Spin up the data migration task, perform the full load into the S3 bucket, and enable change data capture with replication ongoing via a Lambda-triggered Glue job.

Explore why MongoDB, a NoSQL database, is in high demand for web and mobile apps, with freelancing opportunities and strong industry relevance.

Meet instructor Muhammad Ahmed, a cloud and big data engineer who brings experience with databases, data migration, cloud deployment, and DevOps to help you master big data skills, including MongoDB.

Explore how the Udemy review system works, encouraging you to assess the remaining sections and real-world concepts covered, and rate honestly if you think the content is five-star material.

Learn how NoSQL databases use flexible key-value pairs instead of fixed columns to store employee data and dependents. See how this approach reduces joins and enables document-based storage in MongoDB.

Learn to insert multiple documents into a MongoDB collection in one go using insertMany, compare with single inserts, and see how IDs are returned for bulk operations.

Execute basic update operations on documents by updating quantity to 204 for abc 1 2 3 and setting the order to 50 where quantity is 4.5, using the sample data.

Learn how to perform update operations in MongoDB by updating quantities and nested metrics ratings, while navigating document structure and common pitfalls of updating multiple documents.

Use filters to read documents in MongoDB by criteria, such as name Emmett or age, returning only matching records; empty criteria returns all records.

Perform basic read operations to retrieve documents from a collection using find and pretty for formatted output, and filter by fields such as French, science, and history marks.

Learn to read documents from a collection using a read operation: print all documents, format with pretty, and filter by quantity and metrics.ratings equals 3.5 using a find condition.

Learn how to perform delete operations in a database using empty criteria or specific filters, and delete by id by supplying the full object.

Practice basic delete operations by writing queries to remove documents with nine marks in French and those named John, then delete all remaining documents.

Demonstrate deleting documents by criteria in MongoDB, such as nine marks in French or specific student names, and clearing a collection by using an empty query parameter.

Master the basics of delete operations and preview MongoDB operators, as the course prepares you for the upcoming MongoDB module.

Discover the MongoDB $in operator for query and projection, using a single condition to find documents with quantity matching values like 15, 20, or 25.

Explore the $nin operator, the inverse of $in, and learn to filter documents by not in a specified list, applying multiple not-in criteria to inventory data.

Explore evaluation query operators with the expression operator to compare spent and budget. Use the $expr operator in db.collection.find, referencing $spent and $budget with $gt, $lt, $gte, $lte, or $eq.

Learn to use MongoDB's text operator on indexed fields, including phrase searches, case sensitivity, and negation, with text indexes on the article collection.

Explore the MongoDB $all operator to filter documents by array contents. Learn how to require all specified elements in fields like tags and quantity.

Explore the $size operator in MongoDB within the realm of query and projection operators, filtering documents by array length using inventory data to query tags and quantity.

Create a MongoDB collection, insert records, and use the $eq operator to filter documents by quantity and nested item codes in tags arrays.

Create a collection, load data, and query documents using find and pretty. Learn to use the $gte operator to filter by quantity, including nested item fields accessed via dot.

Explore MongoDB query and projection operators with the $in operator, filtering documents by quantity, nested item fields, and codes. Learn practical examples involving arrays and tags to refine results.

Master the less than ($lt) operator in a quiz that filters documents by quantity under 15 and size under 12, reinforcing query and projection concepts.

Explore query and projection operators through a practical quiz, filtering documents by quantity not equal to 20, equal to 5, and not containing the value five.

Explore querying a MongoDB collection with the $ne operator to filter documents by quantity not equal to 20 and by nested values not equal to 10, with projection.

Practice using the not in operator in a quiz on query and projection with $nin, filtering documents by quantity values, tags, and names.

Explore using the not in ($nin) operator in MongoDB to filter documents by name not in a given set, using the Mongo Shell find.

Use the not operator with query and projection operators to combine conditions, such as not one to three, quantity greater than 15, and tags do not contain a or b.

Explore query and projection operators through an exists-based quiz, filtering documents by quantity and field presence to practice building precise database queries.

Explore query and projection operators, especially $exists, to filter documents by field presence and values, using the and operator. Lecture walks through creating a collection, inserting data, and applying queries.