Snowpark : Data engineering with Snowflake.

Initialize a Snowpark session and create a dataframe with session.createDataFrame using a list; Snowflake infers data types when no schema is provided, performing automatic typecasting, while explicit schema handles dates.

Learn how a data frame translates to a SQL query, leverage query cache and temporary tables to reuse results, and cache data frame results for improved performance in Snowflake.

Apply schema explicitly in Snowpark to control data types, compare implicit inference with explicit schema, and see date columns convert to date, highlighting improved performance.

Compare table type and data frame type in Snowpark by creating data frames from data elements and tables, and observe that delete, merge, and update exist on the table type.

Create a Snowpark dataframe from a pandas dataframe and observe that Snowpark creates a temporary background table, linking it to the dataframe and enabling show-based verification.

Learn how to read snowflake tables in snowpark using table and sql methods, understand show for limits, and use cache result wisely to balance performance, memory, and data volume.

Read CSV data from S3 using Snowpark, creating a session and a schema, handling bad records with on error continue, then cache results to optimize performance.

Read json data from s3 into a snowpark dataframe using an external stage and json file format, then parse with select expression. Avoid direct cache on semi-structured reads.

Explore writing transformed data back to a Snowflake table with save as table, using overwrite and append modes, and understand table creation and potential data auditing limits.

Learn how to write json data from s3 to a snowflake table with snowpark, using column aliasing to fix dollar one names, and compare append versus recreate strategies for ingestion.

Read CSV data from S3, apply schema to a data frame, ignore bad records, and write to Snowflake via a temporary table, noting CSV writes use copy behind the scenes.

Learn how to copy data into a Snowflake table using the copy into table method from external locations, compare it to the write method, and apply on-load transformations.

Learn how to compute min, max, and average account balances using Snowpark's aggregate method, exploring dictionary and list syntax and the role of input parameters.

Learn how to identify duplicate records using group by and count in Snowpark, building and aggregating data frames to filter results efficiently.

Use Snowpark to find the minimum account balance in the supplier table and identify the corresponding supplier, via a two-step process of collecting results and aliasing min.

Rewrite the supplier query with a window function in Snowpark to rank suppliers by account balance ascending and filter rank one, avoiding the collect approach.

Explore how Snowpark supports all database join types, including inner join, outer join, and full outer join, with practical examples using customer, order, and line item tables.

Learn to implement the in clause in Snowpark to feed the inner query results into the outer query, filter by nation key 17, and validate against the supplier table.

Build a simple data pipeline with Snowpark using data frames and transformations, plus reusable functions and configuration files to load data from AWS S3 into Snowflake.

Fork the repository, enable code spaces, and set up a conda environment with snowpark packages and the Python interpreter in Visual Studio Code to run the exercises.

Develop a reusable snowpark session method that captures session id, current user, warehouse, and role, and stores these metrics in a Snowflake session audit table.

Build a generic code library for Snowflake connections by structuring a reusable snow connection component, importing shared code, and capturing session metrics for auditing.

Build a generic copy-into-table workflow that copies CSV data from a staging area to a Snowflake table using a config file, recording rejected records and metadata.

Modularize the data pipeline by moving connection parameters, target table details, and schema into separate config and schema folders, and read them in Python to cleanly manage Snowflake operations.

Collect the rejected records after a Snowflake copy by using a generic code library and a config file to store them in the rejected table.

Learn how to copy semi-structured data from a source to Snowflake, parse with Avro, stage temporarily, and use on error continue to safely load and reject bad records.

Learn to copy semi-structured Avro data from S3 into a Snowflake table by reading Avro files, configuring connection and schema, loading to a temporary stage, and using the copy command.

Map specific source columns to the target table by selecting only needed fields and renaming them, such as user_id and user_email, with a dedicated mapping method.

Map columns is a method that takes a data frame and a config map, removes quotes, drops unused columns, and renames to targets. This supports copying into table semi-structured data.

Create and reuse modular data pipeline methods to move CSV data from S3 to Snowflake, handle semi-structured data, and audit loads, while documenting and standardizing a project framework.