Data Engineering with Spark Databricks Delta Lake Lakehouse

Discover Databricks free edition, a unified data and AI platform on Apache Spark with serverless compute, Python support, notebooks, and end-to-end data pipelines.

Explore the complete data workflow in Databricks using Unity Catalog and Delta tables: ingest, read, transform, and write data into Delta tables with ACID transactions and time travel.

Explore the Databricks file system (dbfs) and dbutils, work with free sample datasets, and read files programmatically with Spark from dbfs using magic commands and Spark CSV reads.

Explore Unity Catalog volumes as governed storage for files, enabling read and write, access control, audit, and lineage, with volumes that replace public dbfs paths for production data engineering.

Load the Titanic dataset into a dataframe, filter it, and save as a delta table using Databricks JNI; use generative AI prompts for code and natural language queries.

Explore the Databricks generative AI feature for rapid data engineering, generating and running code, displaying the schema, and storing results in a delta table, all without writing code.

Explore interface differences between Databricks community edition and free edition, including notebooks, file import, and serverless compute versus cluster creation, and note Scala availability.

Explore PySpark transformations and actions on a diamonds dataset, including filter, group by, and order by, while understanding spark core, Spark SQL, dataframes, immutability, and lazy evaluation.

You run a Databricks notebook in Scala, switch between Python and Scala in the same notebook, and use explicit declarations with val or var to perform DataFrame transformations.

Learn the basics of Python and Scala for data engineering, covering data types, loops, lists, and simple functions. Create a notebook to write and run code in Python and Scala.

Explore Spark UDFs to register custom functions and apply transformations in SQL or DataFrame, with examples of concatenation, lambda-based UDFs, and reversing strings on New York taxi data in Databricks.

Learn how to read data from delta tables, join customer and transaction datasets with DataFrame APIs and Spark SQL, and perform country-level aggregations in Databricks.

Explore a range of Spark join operations on customer and transaction dataframes built from delta tables, including inner, left, right, full outer, left semi, and left anti joins.

Merge customer and transaction data to create an insights dataset, then transform and query it at scale using Apache Spark and Delta tables in the Databricks Lakehouse.

Understand transactional databases, data warehouses, data lake, and the lakehouse, with emphasis on ETL, data governance, security, and cloud-based scalability for real-time and historical analysis.

Explore how Databricks Lakehouse combines Apache Spark and Delta Lake to deliver a scalable, ACID transactional data lakehouse with Delta engine, SQL access, and seamless cloud storage integration.

Explore delta tables as data lake components, storing parquet data accessible via SQL or spark, with versioning, time travel, and ACID transactions, while the delta log records changes.

Explore storing data in a delta table with Databricks, perform transformations, and use delta time travel to query, restore, and audit table versions across Spark SQL.

Compare spark sql and databricks sql, create a delta table from diamonds, and use describe detail and describe table to inspect schema and results.

Discover how Databricks Delta table caching speeds read-heavy analytics by keeping Delta data in memory, enabling faster reads for large datasets through explicit cluster caching configuration and demonstrations.

Explore delta table partitioning to speed queries by organizing data into partitions by payment type, reducing scan data and improving performance, with data stored in separate parquet files.

Apply Delta table z-ordering (g ordering) using the optimize command to sort by chosen columns, such as payment type, improving range query performance and reducing disk reads.

Create an AWS account by registering with email, setting a root user and account name, verifying identity, adding payment details, and signing in to access AWS services.

Set up a Databricks account on AWS via marketplace, create a workspace and compute cluster, and manage a 14-day free trial while monitoring billing to avoid charges.

Learn to run notebooks on an AWS Databricks account by creating and attaching a single-node cluster, then execute Python, Scala, or SQL notebooks and work with DBFS and Delta tables.

Build an etl pipeline with delta live tables in Databricks, applying a declarative bronze–silver–gold layering, with data cleaning, aggregation, and transitions between layers.

Learn to manage who can access AWS resources using IAM, create users and groups, apply least privilege through policies, and secure accounts with MFA and careful root access.

Automate and scale data integration with AWS Glue, extracting, transforming, and loading diverse sources into a data warehouse such as Amazon Redshift, with crawlers, schema management, and ML-assisted deduplication.

Explore creating a data catalog in Glue using a crawler to map S3 data to a Glue table and query results with Athena.

Run an AWS Glue Studio ETL job to read bank prospect data from the data catalog, drop the purchased field, and store the CSV output in an S3 bucket.

Explore Amazon EventBridge, a serverless event bus for real-time, decoupled communication using publish-subscribe patterns, rules, and event transformations to multiple targets.

Set up an EventBridge rule to capture S3 object created events and trigger an SNS topic notification, delivered via email, demonstrating AWS services integration.

Discover how AWS Step Functions coordinate serverless workflows with a visual interface, using state machines to orchestrate Lambda, Glue, and other services, enabling branching and parallel processing.

Orchestrate three Lambda functions in a Step Functions state machine to add, multiply, and subtract, with a visual editor showcasing the workflow and successful execution results.

Build an automated data pipeline with AWS Glue, Athena, AWS Lambda, EventBridge, and AWS Step Functions to orchestrate and trigger ETL jobs from S3 events.

Explore real-time data processing with Amazon Kinesis Data Streams, including producers, shards, and partition keys, to enable immediate analytics, dashboards, and multi-application data processing.

Learn to store and retrieve records in a Kinesis data stream using the AWS CLI, including installation, configuration, stream operations, and cleanup.

Learn to write and read real-time weather data on an AWS Kinesis stream using Python and Boto3 in Colab, including producer and consumer scripts with shard iterators and put_record/get_records.

Build an end-to-end real-time data pipeline by creating an AWS Lambda that processes weather data from a Kinesis stream and writes JSON records to S3.

Master the foundations of AWS EC2 for Amazon EMR, launching an Amazon Linux instance, configuring key pairs and security groups, and accessing it via EC2 Instance Connect or Cloud Shell.

Build a data pipeline on Amazon EMR to run a Spark transformation with PySpark, reading bank prospects data from S3, filtering null values, and saving the cleaned output back to S3.

Explore Amazon Redshift, a petabyte-scale data warehouse service on AWS, with columnar storage, massively parallel processing, standard SQL, and integrations with S3, Glue, and SageMaker for analytics.

Create a data warehouse on Amazon Redshift serverless, configure namespace and workgroup, load S3 data with copy commands, and run queries joining customers, orders, and order details to gain insights.

Explore no-code data transformation with AWS Glue DataBrew, a visual, serverless tool for data cleaning, standardization, and anomaly filtering using recipes and automated jobs.

Thank you for enrolling in this course; access resources, exclusive Udemy coupons, and subscribe to our YouTube channel for educational blogs to support your learning journey.