AWS Glue : A Comprehensive Guide for Beginners

Explore the core components of AWS Glue, including data catalog, Glue job, and crawler, to build a solid foundation for beginner-level Glue basics and hands-on practice.

Explore the AWS Glue data catalog as the central repository for data assets, linking sources and ETL targets in metadata tables, populated by crawlers and classifiers from data stores.

Explore how databases organize metadata tables in the AWS Glue data catalog and how each database maps to tables across data sources like an e-commerce store on RDS.

Explore AWS glue data catalog tables and their metadata for ETL sources, including how crawlers determine schema for S3 csv data with columns and types like BigInt and string.

Discover how AWS Glue crawlers automatically classify data, derive schemas, and create or update Glue tables and partitions in the data catalog using built-in or custom classifiers.

Discover AWS Glue jobs for ETL by defining sources, transformations, and targets in a data catalog. Learn how serverless execution, schema updates, and loading transformed data enable data processing workflows.

Get started with a sample ETL job using AWS Glue, covering prerequisites such as an AWS user and IAM role, and introducing the core components discussed in this course.

Discover how to set up an AWS account from signup to the AWS Management Console, including email verification, billing setup, identity verification, and choosing a support plan.

learn how to create an iam user and the aws glue service role, configure s3 permissions, and set up access for glue etl tasks.

Develop a small ETL with AWS Glue by loading flight data from S3, creating a crawler and data catalog, and mapping a 64-column schema to a source table.

Create sample etl job in aws glue visual etl using flight source data, drop fields to seven columns, change schema to timestamp, preview results, and export as csv to s3.

Run and validate an AWS Glue ETL job from the AWS console, then verify in S3 that the transformed data has reduced columns, renamed fields, and departure and arrival timestamps.

Explore AWS Glue data discovery features to organize data and create and manage catalog components such as databases, tables, and crawlers.

Explore AWS Glue crawler data stores across native, JDBC, and MongoDB clients, including MongoDB Atlas, DocumentDB, and databases like Redshift and Snowflake, plus Delta Lake, Apache Iceberg, and Apache Hudi.

Master AWS Glue connections to access data sources from databases to S3, configure JDBC and MongoDB connections, and manage credentials via Secrets Manager.

Demonstrates creating an AWS Glue connection to a MySQL RDS inside a VPC, a glue network interface, a VPC endpoint for S3, and testing the JDBC connection with IAM role.

Discover AWS Glue built-in classifiers for databases via JDBC, NoSQL like DynamoDB and MongoDB, and formats such as Avro, ORC, Parquet, JSON, XML, and CSV.

Create a custom CSV classifier with a plus delimiter and use it to crawl an S3 dataset in AWS Glue, producing a CSV-classified cc_v2 table.

Discover how AWS Glue table partitions divide data by date or category to speed queries, reduce costs, and enable partition keys, while crawlers ensure consistent schemas, formats, and compression.

Create and map table partitions in AWS Glue using a crawler to catalog S3 partition data, including mon partitions for 12 months of 2016 in CSV format.

Map highly partitioned S3 data to the AWS Glue data catalog via a crawler, generating 367,920 partitions with year, month, day, and hour indexes.

Run an AWS Athena query to count records and sum values for December 2000, and see how partition indexes speed up partitioned data on S3 when partition filtering is enabled.

Generate column statistics in AWS Glue to optimize Athena and Redshift query plans, assess data quality, and improve ETL performance by mapping distinct values, min, max, and nulls.

Demonstrate generating column statistics for the flight-source-db partition_csv table in AWS Glue, selecting two columns, using a 10% sample, and reviewing distinct values, nulls, max, and min.

Explore how to build ETL jobs in AWS Glue using visual ETL, AWS Glue notebooks, script editor, and interactive sessions to streamline data discovery and cataloging.

Use visual ETL, a graphical interface that speeds up defining and editing ETL jobs in AWS Glue, with Spark-based Python or Scala scripts generated from the design.

Explore Apache Spark, a large-scale data analytics engine powering AWS Glue, enabling data engineering, data science, and machine learning. Process batch and real-time data with SQL queries and built-in libraries.

Explore AWS Glue built-in transforms like drop fields, change schema, select fields, rename field, and join that convert input data into dynamic frames. Learn how union, split fields, and fill missing values use machine learning to complete data.

Create a custom transform in AWS Glue Studio with a Python function to remove canceled flights from a dataset, then package and deploy it for use in an ETL job.

Explore the AWS Glue Script Editor in Glue Studio to write and manage ETL scripts in the console, leveraging Python or Scala with Spark, Glue, or Ray.

Explore the key Apache Spark APIs, including Spark SQL, the DataSet API, and the DataFrame API, and see how PySpark enables distributed data processing with joins, filters, and aggregates.

Explore AWS Glue PySpark extensions, including the DynamicFrame abstraction over Spark SQL DataFrame, dynamic schema inference, and transforms such as apply_mapping and drop_fields, via GlueContext.

Demonstrates building a spark-based ETL in AWS Glue Script Editor, loading sample flights from S3, filtering cancelled records, and writing the results to S3 as CSV.

Discover the python shell engine in AWS Glue, a non-distributed, single-node option for lightweight python scripts performing data cleansing, transformations, API calls, and simple data movement with boto3 and pandas.

Demonstrates building a Python shell ETL job in AWS Glue to filter canceled flights from S3 CSV data using pandas and boto3, and export to S3.

explore AWS Glue interactive sessions to run transformations and see results. leverage the instant feedback loop with Jupyter Notebooks and Apache Spark for large scale data processing and ETL.

Explore what Jupyter notebooks are and how they enable live code, visualizations, and narrative text in interactive computing, with workflows for AWS Glue and data science.

Explore AWS Glue interactive sessions, a serverless spark runtime for real-time code execution in notebooks, SageMaker, and IDEs, with guidance on security and ipynb conversion.

Create an AWS Glue Studio notebook ETL job using spark python or ray python, load a S3 bucket CSV, and convert between spark and pandas dataframes to run the job.

Enable a local interactive session by setting up Python, installing Glue kernels, and configuring AWS credentials in a Jupyter notebook to build and run Spark dataframes from CSV data.

Discover how magic commands configure AWS Glue jobs, including idle_timeout, glue_version, worker_type, number_of_workers, profiles, regions, session controls, and sql execution.

Explore how AWS Glue streaming enables near real-time data processing and analysis across IoT, fraud detection, social media, and clickstream, powered by Spark Streaming with serverless auto-scaling.

Provision resources with an updated CloudFormation template, including a Kinesis data stream, data generator, S3 bucket, and Glue role; generate about 2000 records and configure the Glue streaming job.

Demonstrates configuring an AWS Glue streaming job from a Kinesis stream, previewing data, renaming columns with a change schema, and writing csv outputs to s3 while processing records.

Learn how Ray offers a simple, pythonic API to scale Python code across a distributed cluster for machine learning and data processing, with remote functions, actors, and Kubernetes cloud integration.

Learn how aws glue enables ray framework for developing and running your ray jobs and scripts using script editor or notebook sessions.

Demonstrates creating a sample etl job with Ray on AWS Glue 4, reading a csv from s3, adding tip_rate, dropping columns, and writing parquet output back to s3.

Explore a Ray-based AWS Glue notebook that reads parquet data from a previous ETL job, inspects the schema, adds tip_rate, drops columns, and validates sample records in Glue Studio.