Polars for Data Engineering: Faster DataFrames in Python

Explore Polars series, a one-dimensional data structure with a single data type, create and print a series with strings, integers, and dates, and see how mixed types yield null values.

Create a multidimensional data frame in Polars, building a dataset with serial numbers, names, date of birth, date time, and location, then print the data frame.

Explore viewing data in Polars data frame using head, tail, sample, and describe to display records and summarize statistics such as count, mean, std, min, max, and percent values.

Master Polars select expressions to retrieve and exclude columns, using name or data type filters with contains, matches, and regular expressions.

Build and manipulate a Polars data frame using numpy-generated records, perform arithmetic operations, apply selection and comparison (less than, greater than), and use pipe expressions.

Learn to rename and alter Polars dataframes by converting column names to uppercase, prefixing or suffixing names, and adding new prefixed columns using dot name, dot map, and lambda.

Explore handling nulls in Polars by creating a data frame with missing values, counting nulls, checking is_null, and filling nulls with strategies like forward, backward, minimum, maximum, and zero.

Use Polars to apply filter and transformations, select all columns, and filter by country code equals us; print the resulting data frame showing three entries.

Learn to sort records in Polars data frames by salary and country code, using select, sort by, and handle ascending, descending, and nulls last.

Import Polars, load and inspect the dataset, then perform aggregation by counting records and computing min, max, and sum of the salary column, plus retrieving the first value.

Explore advanced aggregations in Polars by grouping by country code and aggregating salary and first name, applying having conditions and filters. Count users per group across US and UK.

Explore joining data frames with polars, including inner, left, outer, cross, semi, antijoin, and asof joins, using customer and order data and time-based matching with tolerance.

Explore concatenation in Polars by combining customers and orders data frames vertically, horizontally, and diagonally, handling column name issues and missing values with new aliases.

Explore pivot operations in Polars for data engineering, using group by one or multiple columns and aggregations like first, sum, min, max, mean, and median to reshape data frames.

Learn to unpivot a Polars data frame from wide to long format using melt, with id columns and resulting variable and value columns.

Learn how to use window functions in Polars to compute counts and maxima by partitions such as country code and salary, including expressions like salary modulo two.

Explore Polars' scan_csv method and the lazy frame, replacing read_csv, using collect and filter, and measure execution time to show faster results with employees.csv.

Convert a data frame to a lazy frame using Polars, and print the results to verify the conversion; import plots to illustrate the data flow.

Explore query planning in polars for data engineering using the explained method to compare lazy frame transformations, including adding a column, uppercasing it, and filtering by country code us.

Graph the query plan with show_graph() after installing Graphviz and Matplotlib and setting the bin path to visualize the plan and its projection and filter steps.

Read and write csv files with Polars, using scan for lazy frames and collect to print the data frame, then write to a target directory and verify the file generation.

Explore reading and writing JSON with Polars, parsing JSON into data frames. Convert to structs and generate JSON files, including newline-delimited data and lazy API concepts using scan.

Leverage polars to read and write parquet files, display data as a data frame, and use lazy frames with the collect method for efficient data retrieval.

Read multiple csv files matching a regex like employees_*.csv using Polars for data engineering, load them into a data frame, and print all retrieved data.

Write the dataset into multiple csv files using a range of five, appending numerical values across files and generating five distinct outputs.

Master parallel processing by reading multiple files from the source with a for loop, then combine them into a single Polars data frame using collect all.

Create a script to read data from a Postgres database using SQLAlchemy by establishing a connection, querying the employees table, and printing the results.

Write data to a database table by importing Polars, create a database URI, load the dataset into the users table, and verify the database connection with a successful load.

Initialize sql context and register datasets to manage sql queries for data frame and lazy frame identifiers. Import Polars and print to confirm setup.

Register data and lazy frames in the Polars sql context, using globals, identifiers, or keyword arguments, and learn to bring pandas frames into sql context and collect lazy results.

Register a data frame, lazy frame, and pandas data frame as polars tables and convert the pandas frame to a polars data frame. Run show tables to display registered tables.

Register a file as a table, import data, and execute sql queries to select all from employees, then print the results to demonstrate using polars dataframes with sql context.

Create a new table from the results of a joined SQL query, naming the table and printing the data; the defined table can be used for further operations.

Learn to use common table expressions with clause in Polars, create a lazy frame, register it in a SQL context, and query where age is greater than 20.

Read flat files directly in Polars without registering them in a SQL context, and execute a direct CSV read and query to print results.