PySpark - Build DataFrames with Python, Apache Spark and SQL

Discover Mammoth Interactive, the online school founded in 2012 by John Baraa, known for in-demand tech skills and helping students land dream jobs through project-based learning.

Learn online effectively through project-based learning with industry leaders, flexible pacing, and cost-effective access that delivers faster, results-oriented outcomes while building a portfolio.

Explore Python basics, from variables, operators, conditionals, and loops to functions and classes and objects, all taught from a beginner-friendly browser-based setup using Google Colab.

Explore Python variables by learning placeholders, data types, and dynamic typing. Identify booleans, integers, floats, and strings, and practice type conversion in code.

Explore Python type conversion through hands-on examples, converting between integers, strings, booleans, and floats, and learn how original variables remain unchanged during conversion.

Master Python operators—from arithmetic and assignment to comparison and logical, with practical examples on numbers and strings, including modulus, floor division, exponentiation, string concatenation, and order of operations.

Explore Python collections, including lists, tuples, dictionaries, and ranges, to store multiple values and explore indexing, mutability, and key-value access.

Master Python lists with creation, indexing, and common operations like append, insert, pop, remove, and clear; also construct and access multidimensional lists with row and column indexing.

Explore tuples in Python and compare them to lists, emphasizing immutability, accessing elements by index, and using tuple methods like count, index, and length to inspect values.

Explore Python dictionaries: create, access, modify, and remove key-value pairs; learn inventory examples, and common dictionary methods like get, pop, and clear.

Master Python ranges by using start, end, and step, reverse ranges, convert to lists, and apply in and not in checks inside for loops.

Explore conditionals in Python by learning how if statements, elif, and else control the program flow, test variable states, and handle complex conditions with and/or and nesting.

Learn Python control flow with if statements, elif and else, and the ternary operator through a simple left-right movement example; explore consecutive, nested, and combined tests using and/or.

Explore Python loops, including while and for loops, with break and continue control flow, using ranges and lists to iterate, exit conditions, and practical game-like examples.

Explore Python functions: define with def, pass parameters, return values, and scope principles. Learn how to implement, call, and reuse functions to write cleaner, modular code.

Explore Python function basics by using parameters and return values, adding default parameters, and implementing bounds checks to move a position from start 0 to end 10.

Explore classes and objects in Python by building a game character, learning how classes define blueprints, instantiate objects, use fields and methods, and apply inheritance and static members.

Learn inheritance by subclassing a game character into a player character in Python, using super to initialize and overriding take damage and check is dead.

Learn how static variables and static methods work in Python, distinguish them from instance members, and manage constants shared across all class instances.

Revisit the Python language basics by reviewing variables, collections, conditionals, loops, functions, and classes, and encourage practice and exploration of libraries like pandas for data analysis and machine learning.

Install pyspark in google colab, import pyspark.sql, and create a spark session with app name getOrCreate to verify. Next, upload a csv data file to have data to work with.

Load a CSV file from Yahoo Finance into Google Colab, convert it to a Spark data frame, and prepare for basic data frame operations.

Perform basic dataframe operations in PySpark by inspecting columns, printing the schema, and displaying rows, then summarize data with describe to reveal count, mean, standard deviation, min, and max.

Format data frame numbers in PySpark by importing format_number, casting to float, and rounding to two decimals for columns like open, high, low, close, and volume, then display the results.

Create a new data frame with an open to volume ratio, then sort by high values and compute mean, max, and min on stock data.

Learn to perform dataframe queries with PySpark using filters to count results, compare conditions such as volume under two million versus above two million, and produce concise results and tables.

Create a Spark session and register a temp view to run SQL queries with Spark. Practice selects, counts, filters, and averages on stock data using Spark SQL functions.