Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Deep-Dive in DeltaLake using PySpark in Databricks
Rating: 3.8 out of 5(12 ratings)
148 students

Deep-Dive in DeltaLake using PySpark in Databricks

Unlock the Power of Delta Lake: Master Databricks and Revolutionize Data Management in this Comprehensive Course
Created bySagar Prajapati
Last updated 8/2023
English
English [Auto],

What you'll learn

  • Understand the power of Delta table in Apache Spark
  • Build Delta Lakehouse using Databricks
  • Explore advanced features of Delta Lake, such as schema evolution and time travel
  • Learn to use Databricks effectively for data processing and analysis
  • Understand End to End use of Delta tables in Apache Spark
  • Hands on practice with Delta Lake
  • Learn how to leverage the power of Delta Lake with a Spark Environment!

Explore related topics

Course content

1 section • 21 lectures • 3h 53m total length

  • Introduction and Syllabus4:28

    Discover Delta Lake architecture and Delta tables in Azure Databricks using PySpark and Spark SQL, covering create, read, write, update, delete, overwrite, vacuum, schema evolution, and cloning, with end-to-end project.

  • Architecture and Introduction of Delta-Lake14:00

    Explore delta lake architecture with bronze, silver, and gold layers, detailing ingestion, data refinement, and business logic on a lakehouse with acid properties, schema enforcement, and time travel.

  • How Delta table differs from Normal tables13:48

    Explore how delta tables differ from normal tables in Delta Lake, including managed and external tables, transaction logs, and updatable data using PySpark in Databricks.

  • Create a Delta table22:11

    Create delta tables in Databricks using SQL and PySpark, including both managed and external tables, with partitioning by department and data saved in Delta Lake.

  • Generate Column in Delta table14:04

    Explore generate columns in delta lake with PySpark in Databricks, learn to create a new column derived from existing data using generated always as with constraints and simple casts.

  • Read a Delta table4:28

    Learn to read delta tables in Databricks using SQL and PySpark. Read from a database, external locations, and delta formats, and convert results to a data frame for analysis.

  • Write to a Delta table9:05

    Explore writing data into a Delta table with append and overwrite options using SQL and PySpark. Learn partitioning, save modes, and saving to a Delta Lake location or table.

  • ReplaceWhere while writing to Delta table10:33

    Explore replaceWhere in Delta Lake to selectively overwrite partitions on a Delta table using PySpark and SQL, avoiding full data overwrite.

  • Delete a Delta table6:16

    Delete data from delta tables in Databricks using SQL and PySpark, including row-level deletes with where clauses and conditional filters, and verify results.

  • Update a Delta table10:03

    Learn how to update a delta table using SQL (spark sql) and PySpark in Delta Lake, including updating by id and setting the name.

  • Upsert or Merge Statment in Delta tables21:08

    Learn how to perform upsert operations in delta tables using merge statements in Databricks, enabling incremental loads and slowly changing dimensions with PySpark and SQL.

  • Understand transactional logs in _delta_log folder14:49

    Explore how delta log stores transaction history in the delta log folder, including json and crc files, metadata and commit info, and how checkpoints optimize reads for time travel.

  • How to go to time travel in Delta table using History18:21
  • Restore Delta table to previous version6:37

    Explore restoring a Delta Lake table to a previous version or older snapshot in Databricks using restore and timestamp as of, with insights into history, versioning, and part files.

  • How to add constraint in Delta table6:52

    Explore Delta Lake constraints in Databricks with PySpark, focusing on not null and check constraints, and how to add or drop them via alter table and change column.

  • How to add user meta data information in a Delta table9:01

    Learn to add user defined metadata to a delta table using Python or SQL, set delta commit info, and verify via describe history in Databricks.

  • Schema evolution and enforcement in Delta table12:19

    Explore schema evolution in delta lake tables by updating schemas with overwrite and merge strategies, handling append versus overwrite, and enabling auto merge for seamless changes.

  • Shallow and Deep Clone of Delta table9:00

    Explore shallow and deep clone in delta lake using PySpark in Databricks, distinguishing metadata-only copies from full data and delta log duplication for dev environments.

  • Addition on Deep and Shallow Clone of Delta table7:41

    Explore deep and shallow cloning of delta tables in delta lake with PySpark on Databricks, including version and timestamp cloning and replace options.

  • How to enable Change Data Feed in Delta table10:51

    Learn how to enable change data feed on a delta table to record row-level insert, update, and delete events with commit versions, and query changes via the table_changes function.

  • Reduce small file issue using optimize7:27

    use delta lake's optimize method to group small delta table files into larger ones, reducing file count and speeding reads, with optional targeting by id or partition.

Requirements

  • Basic Databricks and PySpark Knowlege needed
  • Must have experienced in SQL and Python
  • Willing to learn new skills

Description

This is an immersive course that provides a comprehensive understanding of Delta Lake, a powerful open-source storage layer for big data processing, and how to leverage it using Databricks. With hands-on experience and a step-by-step approach, this course explores the core concepts, architecture, and best practices of Delta Lake. Throughout the course, you will gain valuable insights into data lakes, data ingestion, data management, and data quality. You will learn the advanced capabilities of Delta Lake, including schema evolution, transactional writes, asset management, and time travel. Moreover, this course covers how to integrate Delta Lake with Databricks, a cloud-based platform for data engineering and analytics. You will witness the seamless integration of Delta Lake with Databricks, empowering you to perform analytics, data engineering, and machine learning projects efficiently using these technologies. To enhance your learning experience, this course also includes an end-to-end project where you will apply the acquired knowledge to build a real-world data solution. You will design a data pipeline, perform data ingestion, transform data using Delta Lake, conduct analytics, and visualize the results. This hands-on project will solidify your understanding and provide you with practical skills applicable to various data-driven projects. By the end of this course, you will be equipped with the expertise to leverage the power of Delta Lake using Databricks and successfully implement scalable and reliable data solutions. Whether you are a data engineer, data scientist, or data analyst, this course offers immense value in advancing your big data skills and accelerating your career in the field of data engineering and analytics.

Who this course is for:

  • Data Engineer who wants to switch to Azure Big Data Engineer
  • Beginner Apache Spark Developer
  • Data Analyst

Report abuse