Prophecy Data Transformation Copilot for Data Engineering

Explore the evolution of etl from mainframes to cloud data warehouses, from scripts to visual tools, and the rise of distributed engines with prophecy.

Scale cloud-based data transformation to many users and standardized pipelines, using active metadata, large language models, and robust deployment with observability and data quality controls.

Prophecy fulfills ideal data transformation requirements using a cloud transpiler to migrate legacy etl code. It modernizes code and imports frameworks into packages for banks and hyperscalers.

Discover how the data lakehouse merges lake and warehouse benefits, with raw/bronze, silver, and gold layers enabling self-service analytics and machine learning from centralized object storage.

Andreas Kretz introduces Apache Spark concepts, architecture, and the need for distributed systems, then shows prophecy’s low-code gems for building scalable data pipelines on Spark and Databricks, including lakehouse governance.

Explore how Apache Spark uses a driver program, Spark context, executors, and cluster manager to enable in-memory, parallel processing with partitioning and map, shuffle, reduce phases.

Explore spark core concepts such as rdds and dags, spark sql with dataframes and datasets, and the Python, Scala, Java, and R APIs, plus lazy transformations and prophecy gems.

From Apache Spark to Databricks, learn why modern platforms favor a cloud lakehouse model with notebooks, serverless sql warehouses, and open spark code for flexible pipelines.

Explore the lakehouse architecture, leveraging Unity Catalog to govern bronze, silver, and gold data layers, with data lineage, metastore, and robust security and optimization features.

Optimize Spark data pipelines by tuning partitioning for parallelism and minimizing shuffle, apply early filtering, and use columnar formats like Parquet or ORC, with caching for repeated access.

Prophecy translates visual pipelines into open source code and runs them on an execution engine like Databricks, using fabrics tied to a workspace URL and access token.

Create a prophecy project to manage Spark code, outlining pipelines, data sets, and jobs stored as code in git, with configurable root directory, default branch, connections, repositories, and PySpark dependencies.

Create and test a batch data pipeline on the pipeline canvas, attach a Databricks cluster, select Salesforce sources, and enrich accounts with opportunities using built-in gems.

Explore code view to see prophecy's conversion of visual pipelines into open source PySpark. Build an aggregate by account, counting opportunities and summing amounts and revenue in the last quarter.

Prophecy lets you deploy production pipelines by scheduling jobs with Databricks workflow or Airflow through a low-code interface. Commit, release, and deploy to Databricks, syncing schedules and enabling email notifications.

Prophecy users integrate with git and Databricks permissions to manage pipelines, datasets, and clusters; teams own projects, fabrics, and jobs, with personal projects and sharing via package hub.

Explore Rainforest's data processing layer, learn how to ingest raw data from various formats and scenarios, and master partitioning target delta tables to optimize lake house workloads.

Read raw object-store data, including csv, parquet, and json, and ingest into the bronze layer on S3 using partitioned folders, 128-512 MB files, and splittable formats.

Create a dataset called members from a csv in the Databricks file system, infer schema from the header, preview the data, and finalize the source using the default configurations.

Read parquet data with evolving schemas and merge them to form a unified data set across dates, demonstrating the merge schema option and handling nulls.

Handle corrupt and malformed json records from object stores by using permissive, drop malformed, or fail fast modes, infer schema, and ignore null fields to ensure reliable data loading.

Explore options to handle corrupt and malformed records in data transformation, including drop malformed, permissive fixes for dollars to cents, enforce schema, and fail fast on errors.

Learn to work with source data schemas and delimeters by inferring csv schemas with and without headers, correcting tab delimiters, and labeling columns like customer id and date.

Connect to Unity Catalog to read catalog tables as sources, add them to the canvas, and read, parse, and process raw data to enrich datasets after the bronze layer.

Write bronze-layer delta tables from raw data in the data lake using a data target gem and Unity Catalog. Overwrite to store the product line data as a catalog table.

Partition data when writing to a delta table to boost query and write performance by partitioning on date introduced, balancing partition size, cardinality, and data user requirements.

Learn to read and write data with prophecy sources and targets, infer schemas and handle corrupted json and csv formats, enabling a data lakehouse bronze layer with merged schema.

Explore the medallion architecture in a lakehouse, moving data from bronze to silver to gold, using Prophecy to design pipelines that transform raw data into analytics-ready tables.

Explain the medallion architecture: bronze stores raw, unprocessed data in the object store, silver cleans and deduplicates for querying, and gold provides clean aggregates for dashboards and data marts.

Implement bronze layer by partitioning reads for daily loads from object storage, using a dt parameter; write to a bronze delta table by overwriting the date partition.

Change the parameter to the next day and re-run the pipeline. Overwrite only the January 2nd partition from object storage to avoid duplicates and ensure daily data is correct.

Explore data models in a data lake house, from logical to physical designs, including star schema, snowflake, data vault, and map bronze data to silver and gold layers for insights.

Load JSON sales events into the bronze layer and write to delta tables partitioned by sale date using the overwrite partition predicate, then read table to verify January 2nd data.

Implement simple scd2 for a bronze table by tracking a single customer’s history with valid from and valid to, using is original and is current flags.

Execute a bulk scd type 2 load by deduplicating changes and computing valid from and valid to with window functions (lead/lag) partitioned by customerid, then write to a delta table.

Learn how delta table data versioning complements SCD type two to track historical records and schema changes using delta table history, read versions, and read timestamps.

Navigate incompatible schemas by casting campaign id from long to string, and learn how overwrite mode and schema overwrite options interact with delta history and time travel constraints.

Use the Delta Table operations gem to restore a previous version of a table, review compatible and incompatible changes, and publish a new version while preserving history.

Transform the bronze to silver layer by matching, merging, and cleansing data into delta tables, using the flattened schema gem, join gem, and aggregate gem to support a star schema.

Build a data model in a star schema by integrating fact and dimension tables, flattening nested product data, and enriching sales with customer and product details in the silver layer.

Implement SCD2 in the silver layer by joining sales, customers, and products on product ID and customer ID, using valid from and valid to to enforce correct historical joins.

Explore how CI/CD automates building, testing, and deploying data pipelines to production on a fabric connected to a Databricks workspace.

Explore advanced integration of prophecy ci cd with external workflows using the prophecy build tool (pbht/pbbt) to build, test, and deploy pipelines in Databricks.

Collaborate with your team to manage changes and resolve code conflicts in pipelines and jobs using a git provider, with project sharing, reviews, and engineering best practices.

Collaborate on a rainforest project by pulling main into the development branch to catch up with another engineer’s feature one pipeline, then review two main commits and prepare a revert.

Learn to revert changes in a data pipeline during development before committing, using reset or pipeline-level revert to undo unit and quality checks and restore the prior pipeline state.

Resolve git conflicts when multiple team members commit by pulling, comparing main and development branches, choosing the right version, and merging to main to keep the pipeline up to date.

Clone an existing Prophecy project to a new repository with proper git permissions and root or subdirectory paths; commit in main before cloning, since only the main branch is copied.

Collaborate effectively within a team to develop data pipelines and jobs, resolve code conflicts, and manage code versions with git, importing and sharing code across projects.