Master Data Engineering using GCP Data Analytics

Set up a data engineering environment for GCP data analytics on Windows or Mac, including VS Code, Python 3.9, path configuration, and cloning the repository to build your workspace.

Set up Visual Studio Code on Windows 11 to use as the primary ide for the course, highlighting cross-language support in Java, Python, and Scala.

Install Python 3.9 on Windows using the 64‑bit installer from python.org, then validate by launching Python and printing hello world; learn to configure PowerShell PATH in the next lecture.

Configure the Windows environment by updating the system path to include Python 3.9 so the Python CLI launches from Windows PowerShell, then validate with Hello World.

Install the python extension and integrate python 3.9 with vscode, set up a workspace, and run hello world in the terminal. Validate integration and adjust font size for python development.

Download the data engineering on gcp repository as a zip, extract it locally, and set up a VS Code workspace to access notes, notebooks, scripts, and data.

Review the data engineering on GCP workspace setup, including markdown materials, notebooks, datasets, apps, and scripts, to configure a VS Code workspace for GCP data analytics services.

Set up a VS Code workspace for the data engineering on GCP folder, ensure notebooks and apps in VS Code, and integrate a Python virtual environment for the project.

Set up a Python 3.9 virtual environment and integrate it with the VS Code workspace using the terminal. Create venv, verify Python 3.9, and select the interpreter to activate it.

Configure the VS Code workspace, create and activate a python 3.9 virtual environment named deg-venv, and select the interpreter from that environment to ensure proper integration.

Set up a robust data engineering environment for GCP data analytics by configuring VS Code, Python 3.9, and a Python virtual environment; validate setup via terminal and Python runs.

Outline prerequisite skills, provide a quick cloud platforms overview, then cover GCP basics, sign up for GCP, and set up the Google Cloud SDK to enable data engineering on GCP.

Identify the prerequisite skills for this course, including required SQL and programming knowledge (Python preferred), with highly desired data engineering and data warehousing fundamentals and command line tool familiarity.

Explore cloud platforms and their core features, including quick setup, marketplace access, pay-as-you-go models, and scalable infrastructure, with emphasis on AWS, Azure, and GCP.

Explore the Google Cloud Platform marketplace of native and third-party services, managed via CLI or the Google Cloud SDK, and learn to select storage, databases, and serverless options.

Sign up for GCP with a valid Google account, claim $300 credits for three months, and set up a project while reviewing billing.

Learn how to create a Google account with a non Gmail id and sign up for Google Cloud Platform, including email and phone verification steps.

Sign up for GCP using a Google account, start free with $300 credit, enter email and password, add card details, and complete setup with data management and analytics in mind.

Activate the $300 GCP credit and use it for up to 90 days, with free-tier limits per service (e.g., BigQuery 1 TB, storage 5 GB) before charges apply.

Learn to manage a GCP project from signup to naming, noting that the project ID cannot be changed after creation, and review billing, credits, and gcloud setup on Windows.

Explore Google Cloud Shell, free for all users, to manage infrastructure and develop applications from any browser, using gcloud, gsutil, and the Cloud SDK, with hands-on project setup in GCP.

Install the Google Cloud SDK on Windows using the Google Cloud CLI installer and PowerShell integration, then validate gcloud and gsutil to develop against GCP services locally, including storage.

Master data engineering with GCP data analytics by installing and initializing the Google Cloud SDK and gcloud CLI, authenticating, selecting a project, and validating access to Google storage with gsutil.

Reinitialize the Google Cloud Shell with your project id using gcloud init, select the tidy fort 361710 project, then run gsutil list to verify buckets.

Get an overview of GCP analytics services, including Cloud Storage for data lakes, Cloud SQL, Dataproc, Databricks, BigQuery, Cloud Functions, and Cloud Composer for data processing at scale.

Get started with data engineering on Google Cloud Platform by setting up the Google Cloud SDK, validating your Google Cloud account, and exploring analytics services.

Explore google cloud storage as a reliable, secure object storage solution, compare gcs with s3 and adls, and review storage classes, redundancy, and transfer options.

Learn to use the Google Cloud Storage web UI to create buckets, manage objects, choose multi-region placement, select storage class, and apply versioning, retention, and encryption.

Create a GCS bucket via the web UI, using multi-region default and standard storage class, review permissions, and set up uploading files with the web UI, later leveraging CLI automation.

Learn to upload folders and files to a GCS bucket via the web UI, including recursive folder uploads, navigating a local data path, and validating bucket contents with gsutil.

Use gsutil to list buckets and objects in gcs via cloud shell, handle authorization prompts, view bucket URLs, and prepare for programmatic object management with Python.

Delete a Google Cloud Storage bucket using the web UI, confirm deletion, and verify removal, then explore bucket management options and object handling in future lectures.

Clone the data repository in the Google Cloud Shell home folder and use gsutil to manage files and buckets in GCS, then explore retail_db and retail_db_json datasets.

Review the data folder in the GitHub repository or zip file, explore datasets like nyse, retail db, and h1b data, and learn to copy them into GCS with gsutil.

Master gsutil to manage google cloud storage buckets and files. Use mb to create and rb to delete buckets, emptying before deletion, and copy local files to GCS.

Copy the local data folder retail_DB into the aidldemo bucket under data using gsutil cp -r, then verify with gsutil ls -R and consider using -m to speed up transfers.

Learn to clean up a GCS bucket with gsutil by removing all objects recursively and then deleting the bucket using rb or rm -r, following a two-step non-development approach.

Practice creating a Google Cloud Storage bucket with gsutil, recursively copy the data/retail_db_json folder into the bucket under data/retail_db_json, and validate the copy.

Learn how to set up a data lake on GCS and manage files with Python, validating the environment, and using key APIs in a notebook-based quickstart.

Set up the google cloud sdk and python libraries in a virtual environment to manage files in gcs, verify gcloud and gsutil, and configure the correct account and project.

Create a new GCS bucket and copy the retail_db dataset into it using gsutil from a notebook, then validate with gsutil ls.

Discover how to manage files in Google Cloud Storage using Python, configure a virtual environment, install the Google Cloud Storage library, and handle credentials for notebook and script usage.

Authenticate external python applications to Google Cloud storage by setting up credentials with gcloud auth application-default login, then create a GCS client to manage buckets and files.

Explore the Google Cloud Storage Python library to manage buckets and blobs, using gsclient to list, create, and get bucket details, and download blobs to local files.

Learn to list buckets and fetch bucket details with a client, using list_buckets and get_bucket in Python, inspect bucket attributes, and manage blobs within the bucket.

Explore managing GCS blobs and files with Python: list blobs in a bucket, create blob objects, and upload or download using upload from file name and download blob to file.

Upload all files and folders from a local data folder to Google Cloud Storage using Python, with glob and os, leveraging the google.cloud.storage module to address the problem statement.

Design and implement a Python-based, data-driven workflow that lists local files with glob, creates blob objects named after each file, and uploads multiple files to Google Cloud Storage with validation.

Review the Get_file_names function to recursively collect data files with glob and os, filter out SQL scripts, and prepare a list for uploading to GCS in parquet format.

Learn to upload files to Google Cloud Storage as blobs using Python, preserving folder hierarchy under the python demo bucket and constructing blob names with split and join.

Validate that the nine local files are uploaded to GCS as blobs by listing local data, using gsutil ls -r, and programmatically listing blobs with Python.

Read and validate data stored in Google Cloud Storage using pandas and GCSFS, leveraging read_csv and optional column names. Write processed data to GCS in parquet format, following notebook examples.

Process local data with pandas, convert to a dataframe, and write to Google Cloud Storage in parquet format using pyarrow and gcsfs, applying a schema and disabling index.

Design a pandas-based workflow to upload multiple local files to GCS by listing files, reading schemas, constructing dataframes, and writing parquet files to GCS with snappy format.

Learn to retrieve all file names from a folder recursively using glob and the os module, filter to include only files, and prepare for uploading these as blobs to GCS.

Explore parquet as a binary, columnar format and learn to extract column names from schemas.json in data/retail_db to align with table definitions and column positions.

Load schemas.json into a Python dict, extract column names by sorting on column position, and validate across datasets. Next lecture covers pandas read and write to GCS.

Upload files to GCS as parquet using pandas, constructing the target path from the dataset name and schemas, and write data frames to snappy parquet with to_parquet.

Validate that parquet files copied to GCS match the source data by reading with pandas read_parquet, comparing shapes and row counts across six datasets.

Explore Google Cloud SQL, a fully managed relational database service for MySQL, PostgreSQL, and SQL Server, emphasizing quick setup, automated backups, high availability, and migrations from on‑prem to cloud.

Set up a Postgres database server using Cloud SQL on Google Cloud Platform by enabling the API and creating a Postgres instance in development with Postgres 14.

Configure network access for Cloud SQL Postgres, validate connectivity with Telnet, and manage authorized networks by adding your IPv4 address and handling IP changes.

learn to install Postgres 14 on Windows 11 by downloading the latest installer, running setup, and verifying the server is running.

Connect to the postgres server on Windows using pgAdmin to manage databases and tables via the GUI, and optionally use the PSQL CLI for operations.

Learn how to validate a Postgres database on Mac using pgAdmin, set a master password, connect to the local server, and run SQL queries against information_schema.tables to verify setup.

Validate that pgAdmin and psql are available on Mac or Windows, enabling local access to the Cloud SQL Postgres server and future connection via these tools.

Connect to a cloud sql postgres database using psql, create the database and user, grant permissions, and run scripts to set up tables and datasets.

Set up and populate Postgres tables in a GCP Cloud SQL environment by running two SQL scripts (create_db_tables_pg.sql and load_db_tables_pg.sql) via psql, creating six tables and loading data.

Validate data in Gcp cloud sql postgres database tables by listing the six created tables and running simple select queries and counts to confirm data for departments, orders, and order_items.

Learn how to connect Python to a Cloud SQL Postgres instance using psycopg2-binary, install via pip in a virtual environment, specify host and port 5432, run queries with a cursor.

Explore integrating GCP Cloud SQL Postgres with pandas to build lightweight data pipelines, using pandas with SQLAlchemy and psycopg2-binary to interact with Postgres for read and write ETL.

Learn to read the orders data from CSV into a pandas dataframe using a defined schema, aliasing pandas as pd, and dynamically derive column names from schemas.json.

Process data using the pandas DataFrame API to group by order date and status, compute counts via agg, and reset the index for loading results into a Postgres database.

Learn how to write a pandas dataframe to a Postgres table using the to_sql API, with options to create or overwrite, configure an SQLAlchemy connection, and validate the write.

Validate data in Postgres databases by loading data with pandas read_sql into a dataframe, query with SQL or read_sql_table, and verify daily_status_count against expectations.

Discover how to use GCP Secret Manager to replace hardcoded credentials, enable the API, create secrets with JSON payloads for database connections, and prepare Python apps to use them.

Configure Google Cloud secret manager access for external applications by assigning the secret manager secret access role via iam, enable the api, and prepare for programmatic secret use.

Install the Google Cloud Secret Manager Python library and verify your environment is set up; then learn to retrieve secret details to connect to the database in upcoming lectures.

Learn how to fetch secret details from Google Cloud Secret Manager using Python, decode the payload data, convert to a dict, and access database credentials for a Postgres connection.

Learn to connect to a postgres database using credentials retrieved from GCP secret manager, building a dynamic connection uri from secret details and avoiding hard-coded credentials with pandas.

Learn how to provision a GCP Cloud SQL Postgres server, create databases and tables, load data, connect with Python and pandas, and stop or start to save compute costs.

Explore Google BigQuery, a cloud-based, fully managed, serverless data warehouse on GCP. Learn to create tables, load data, and enable real-time analytics, ML, and multi-cloud analysis with BigQuery features.

Explore Google BigQuery as a cloud-based, serverless data warehouse on GCP, enabling real-time and predictive analytics, multi-cloud analysis, BigQuery ML, and ELT pipelines with BI Engine.

Learn how to get started with Google BigQuery using the SQL workspace in the Google Cloud console, including data transfers, scheduling queries, analytics hub, and migration tools.

Explore CRUD operations in Google BigQuery through hands-on examples of creating datasets (schemas), a users table with user_id, user_first_name, user_last_name, is_active, and performing insert, update, delete, and the upcoming merge.

Master merge (upsert) in BigQuery by updating and inserting records from a source into a target table using when matched and when not matched conditions.

Create a dataset and an empty orders table in Google BigQuery using the UI, selecting project ID, location, and a four-field schema: order_id integer, order_date timestamp, order_customer_id integer, auto_status string.

Create a BigQuery table from the command line using a create table command with a fully qualified name, defining the schema and data types for order_id, order_date, order_customer_id, and order_status.

Practice creating empty tables in Google BigQuery by adding order_items and products to your dataset, as preparation for loading data with a Python-based approach in the next lecture.

Learn how to load data from files in google cloud storage into bigquery tables, using python to install libraries, stage data in gcs, and run queries for validation.

Learn to load data from GCS to a BigQuery table with Python, using the Google Cloud BigQuery library, creating a client, and running a query.

Load data from GCS files into a BigQuery table using Python, qualifying the table with project id and dataset, and load from URI with a schema in load job config.

Learn to run BigQuery queries from Python to validate loaded data. Create a BigQuery client, run a query job, and read order_id, order_date, order_customer_id, and order_status.

Load data into BigQuery tables using a Python approach from GCS, creating and populating orders, order_items, and products with load job configs and validation.

Drop existing BigQuery tables using the editor’s delete action or a drop table command, using if exists and fully qualified names, before creating tables from GCS files.

Explore external tables in BigQuery that reference GCS files to perform ad hoc analysis without moving data, and use Python pandas for small data before building pipelines to load data.

Create a Google BigQuery external table on GCS files using the BigQuery UI, covering dataset creation, selecting CSV format, defining the schema, and choosing external vs native table behavior.

Create a Google BigQuery external table on GCS files using command; learn the syntax, formats, URI arrays, and optional columns, and validate by querying retail.orders without copying data.

Explore creating external tables in Google BigQuery beyond Google Cloud Storage, linking to sources like Google Drive, Google Bigtable, Amazon S3, and Azure blob storage, with multi-cloud integration.

Create Google BigQuery external tables for orders, items, and products using GCS files; verify results with queries, and choose commands or UI while ensuring external, not native, tables.

Explore the SQL capabilities of Google BigQuery, a GCP data warehouse, including standard SQL features, predefined functions, joins, aggregations, windowing, and JSON processing for data engineering, BI, and AI/ML applications.

Learn to compute daily product revenue in Google BigQuery by inner joining orders and order items, summing order items subtotal, rounding, and sorting by date asc and revenue desc.

Learn to compute daily revenue in Google BigQuery by joining orders and order items, then build monthly cumulative revenue using partition by month and cumulative sum with a CTE.

Explore computing daily product revenue ranks in Google BigQuery using dense_rank and rank. Learn to create views or cte for persistent ranking on partition by date and revenue.

Understand how to compute dense_rank within a date-partitioned view using over and partition by, and filter results by the computed ranks with subqueries in BigQuery.

Explore key Google BigQuery integrations, including GCS data loading and external tables, Cloud SQL and ETL pipelines, and Spark on Dataproc to feed BigQuery-backed applications in Python, Java, or Scala.

Explains how to connect pandas with Google Cloud BigQuery using the pandas-gbq plugin, install required libraries, run queries via read_gbq, and load results into a dataframe, enabling write-back to BigQuery.

Explore how BigQuery runs external queries against Cloud SQL databases, using Postgres and MySQL, to prototype reports, join external data with BigQuery, and support ad hoc analysis.

Set up a Cloud SQL Postgres server, configure BigQuery external queries, validate the retail database and its tables, and use pandas with SQLAlchemy to run queries.

Create external connections in BigQuery to a Cloud SQL Postgres database, then run external queries with the external_query function to explore tables and analyze orders by date.

Learn how to use external connections and queries in BigQuery to pull products from Postgres, join with orders and order items, and compute daily product revenue for reports and dashboards.

Learn to start with GCP Dataproc by enabling the dataproc API, navigating the cloud console, and using clusters, jobs, and workflows for ELT pipelines built on Hadoop, Hive, and Spark.

Set up a single node dataproc cluster for development in the Google Cloud console, choose region us-central1 and a Hadoop-Spark image, and enable Jupyter notebook via the component gateway.

Demonstrate SSH connectivity to the Dataproc master node using the gcloud command to copy keys and connect via the external IP, and plan for a static IP to prevent changes.

Configure a static IP for the Dataproc master node VM by reserving the IP from ephemeral addresses, ensuring it remains static for the cluster, and enabling VS code development.

Connect visual studio code to a dataproc vm via the remote window, using ssh configuration and the remote development extension pack to set up the development environment.

Set up a VS Code workspace on the Dataproc master node by cloning data engineering on GCP repository with Git. Open notebooks, scripts, and datasets to learn Hadoop and SPARC.

Learn how to manage files in Dataproc using HDFS commands and gsutil, exploring HDFS subcommands like ls, mkdir, put, and get within a remote terminal, with GCS interaction.

Recap gsutil for managing GCS buckets and files from the data proc masternode vm, using ls/list, gs:// access, and recursive -R to explore bucket contents, with hdfs data transfers upcoming.

Review local datasets in the data folder, including retail_db, and copy data into HDFS from the Dataproc master node VM using HDFS commands, with sql scripts and schemas.json guidance.

Create the /public folder on the hdfs root, copy the local retail_db folder there with hdfs dfs -put, and verify via hdfs dfs -ls /public.

Practice copying data from GCS into HDFS on Dataproc using gsutil for bucket management and HDFS cp for cross-storage transfers, including recursive copy and delete operations.

Validate pyspark and spark CLIs in a dataproc cluster by creating dataframes for orders and order items, then compute daily product revenue by joining them.

Launch Spark Shell to validate the Spark Scala CLI on Dataproc, create dataframes from orders and order_items, and compute daily revenue with sum, round, and dollar.

Validate Spark SQL CLI on a Dataproc cluster by creating temporary views orders and order_items, joining them to compute daily product revenue, and reviewing view definitions and descriptions.

Explore Google Cloud Dataproc jobs and workflows, including Hadoop jars, Spark SQL, PySpark, Hive, and Pig, and learn to create modular jobs and orchestrated elt data pipelines with workflow templates.

Set up data proc jobs with spark sql using a json dataset in GCS. Copy the json dataset to GCS, validate with gsutil, and explore the retail json dataset.

Learn to run spark sql queries on dataproc against json datasets in GCS using file paths, then save daily status counts back to GCS in json format with insert overwrite.

Launch Dataproc jobs using spark sequel to query, process, and write results, then validate at file and data levels with gsutil and spark sequel, and explore Dataproc workflows.

Modularize spark sql applications for dataproc by building three reusable modules—clean up, json-to-parquet conversion, and daily revenue computation—and orchestrate them in dataproc workflows to process GCS data.

Review spark sql scripts for dataproc jobs and workflows, copy to gcs, perform unit testing or validation on the dataproc cluster, and orchestrate with dataproc workflows in the ui.

Validate Spark SQL scripts via local unit testing, using gsutil and Spark SQL to generate parquet files in a bronze database and validate data with sample queries.

Run a Spark SQL script to convert order items data into Parquet, validate results with unit tests, and verify the order items table and related data in the target database.

Modularize and validate spark sql scripts for dataproc jobs, using orders and order_items in retail_bronze_db. Demonstrate end-to-end validation with spark-sql and gsutil on gcs, confirming daily_product_revenue in retail_gold.db.

Develop a spark-sql cleanup script to drop retail_bronze_DB and retail_gold_DB with cascade, enabling repeated workflows. Validate by running show databases and gsutil ls checks on the airetail bucket.

Copy spark sql scripts from local scripts/daily_product_revenue to airetail gcs bucket using gsutil cp -r, then validate with gsutil ls and prepare to run them from data proc.

Validate spark sql scripts in GCS by enforcing cleanup, file format conversion for orders and order items, and daily product revenue. Use local validation and unit tests.

Examine the limitations of running Spark SQL scripts in Dataproc jobs and workflows, including query text versus file, path handling for GCS/HDFS, and CLI workarounds.

Learn to manage dataproc clusters using gcloud commands, submit dataproc jobs including spark sequel scripts from external clients, and configure regions and workflows for streamlined operations.

Learn to run dataproc jobs with gcloud, submitting spark sequel jobs using -e or --execute with a cluster, including simple queries like select current_date, then monitor in the webui.

Learn to submit data proc jobs using spark sql scripts with gcloud and gsutil, pass parameters correctly, convert data to parquet, and validate results in retail_bronze_db.

Submit spark sql scripts as Dataproc jobs with and without arguments using gcloud, validate results with simple queries, show tables, and review scripts before deletion in the next lecture.

Learn to delete dataproc jobs with gcloud commands, listing jobs, confirming deletions, and deleting one job at a time, with scripts for Linux.

Master dataproc job management with gcloud commands, including submission, maintenance, and instantiating workflows, and prepare for workflow templates and orchestration with tools like airflow.

Explore Dataproc workflow templates in the web UI, configuring templates, clusters with labels, and spark sql jobs with dependencies, then run and review workflows via commands.

Learn to manage Dataproc workflow templates with gcloud commands, attach clusters, add Spark SQL or PySpark jobs with dependencies, and implement a four-job ELT design storing results in GCS.

Create a Dataproc workflow template and attach it to a running cluster using gcloud commands, including listing, creating, and configuring cluster selectors with labels.

Add jobs to dataproc workflow templates with gcloud data proc workflow templates add-job, using spark sql scripts and step id, parameters, and file paths for orchestrated pipelines.

Add four jobs to the Dataproc workflow template using gcloud commands, with dependencies on cleanup before convert orders, convert order items, and daily product revenue, updating bucket names and paths.

Instantiate the dataproc workflow template to run the data pipeline using instantiate or instantiate from file, optionally passing parameters and exporting the workflow to YAML.

Manage Dataproc operations by listing, describing, canceling, and deleting workflow runs, then perform pre-run and post-run validation to ensure clean, accurate data pipelines.

Learn to run and validate an ELT pipeline on Dataproc using gcloud data proc commands, including pre and post run validation, cascade database cleanup, and pipeline review.

Learn to stop, start, or delete a Google Dataproc cluster to stop charges from GCP and retain work on the master node, using gcloud data proc commands.

Explore how Databricks on GCP, a cloud-agnostic big data processing platform available on GCP and Azure, enables orchestrated pipelines with Databricks workflows and GCS integration for end-to-end analytics.

Sign up for databricks on gcp via the gcp console, choose a 14-day trial premium plan, complete registration with google, then access workspaces to use databricks clusters.

Create a Databricks workspace on GCP by configuring the project ID, region US central, quotas, and Google Cloud storage buckets, then connect to set up clusters and deploy jobs.

Spin up a databricks cluster on google cloud platform using the ui, choose a single node with the latest runtime, name it analytics dev cluster, and set a termination time.

Set up a Databricks notebook in your GCP workspace and create a Python notebook in home folder. Use magics like %fs to manage dbfs files and run code with shift-enter.

Explore Databricks on GCP, highlighting cloud-agnostic spark-based processing, GCS integration, purpose-built clusters for data science, engineering, ML, and SQL, with pay-as-you-go pricing and BigQuery BI integrations.

Explore the Databricks architecture and detail control plane, data plane, dbfs, external data sources, and how you manage notebooks, jobs, and clusters via web ui, cli, and rest apis.

Set up the Databricks CLI in a Python 3.9 virtual environment within a VS Code workspace, install databricks-cli, activate dd-venv, and validate the installation with databricks --version and --help.

Explore how to authenticate and use the Databricks CLI to manage workspace components via the control plane, REST APIs, and tokens across cloud platforms.

Configure the Databricks CLI by generating a token and setting the host and a dedicated profile (analytics GCP). Validate with databricks fs ls and troubleshoot misconfigurations or permissions.

Troubleshoot databricks cli by inspecting the .databrickscfg in your home directory, verify host and token, then reconfigure with the databricks configure command.

Master the Databricks CLI to manage clusters, dbfs files via fs, jobs, and secrets, explore the control plane, and use --help and profile options.

Clone the data repository and set up datasets for the course, then prepare to copy the retail_db folder into dbfs using databricks cli.

Copy the retail_db from the local file system to dbfs with Databricks fs cp --recursive, creating the public/retail_db path, then verify with dbfs ls.

Learn to process data in dbfs using Databricks spark sql, employing sql magic %sql to run spark sql queries on csv datasets stored in dbfs on gcs.

Learn to run Spark SQL in Databricks notebooks, ingest data from DBFS retail_db, and compute daily product revenue by joining orders and order_items, using order_date, product_id, and order_item_subtotal.

Create and validate temporary views for orders and order items using Spark SQL with text files in Databricks, then prepare for daily product revenue analysis.

Create a temporary view for order_items using Spark SQL, guided by the data model diagram, and validate it with describe and select to prepare for calculating daily product revenue.

Compute product revenue by joining orders and order_items views in Spark SQL, filtering for complete or closed orders, group by date and product to sum revenue rounded to two decimals.

Write the daily revenue results to dbfs in parquet format using insert overwrite to a specific path, then verify the output by querying and sorting by date and revenue.

Explore PySpark examples on Databricks, using schemas.json to obtain column details for six tables, read data into a DataFrame with the schema, and convert to parquet using those columns.

Learn to extract dataset schemas from a JSON file in dbfs with spark, sort columns by position, derive column names, and apply the schema to read data.

Apply pyspark to read a csv into a dataframe, infer schema from data, and apply column names from schemas.json using spark.read.csv and toDF for an accurate dataframe.

Create a dataframe from csv with a schemas.json schema, then group by order_status, count, and sort by count in descending order using spark APIs and aliasing; prepare for Parquet conversion.

Use a pyspark function to extract schemas from schemas.json, derive column names for datasets like orders and customers, and convert csv data to parquet using those schemas.

Convert csv to parquet by applying schema with pyspark, write data to parquet in dbfs paths using overwrite mode, across all datasets via dynamic base and target directories.

Orchestrate data pipelines using Databricks workflows by building jobs with dependent tasks, configuring notebooks or Python scripts, and triggering executions internally or via external APIs.

Design and run Databricks workflow jobs, configure notebooks, and pass parameters to Python notebooks using dbutils.widgets and get, with text, remove, and the %run magic to orchestrate.

Learn how to pass parameters to SQL based notebooks in Databricks workflows, including using widgets and Python magic to pass and read arguments from external notebooks or jobs.

Create and run your first Databricks job using workflows by adding notebook tasks, configuring clusters, passing parameters, and setting task dependencies.

Learn to pass parameters to Databricks job tasks by defining arguments at task or job level and run with API-driven parameters for notebooks like 03a and 03b.

Create and run an orchestrated Databricks pipeline by wiring notebooks into a job with dependencies and parallel tasks in workflows. Pass runtime parameters and monitor runs, schedule or trigger externally.

Import and run ELT data pipeline notebooks in Databricks by importing the DBC archive, exploring Python and SQL notebooks, and developing a pipeline for daily product revenue.

Orchestrate an ELT pipeline in Databricks to compute daily product revenue, clean the bronze database and parquet folders, and review notebooks with dbutils widgets for dynamic gold data paths.

Review the file format converter pyspark code that reads CSV data, applies schemas from schemas.json, and writes parquet files for orders, with schema lookup and validation via spark read.parquet.

Review two notebooks to build sql logic on parquet files with dynamic external spark tables in a bronze directory, computing daily product revenue by joining orders and order items.

Orchestrate an ELT pipeline in Databricks by cleaning data, converting csv to parquet, creating spark sql tables, and computing daily product revenue using a bronze to gold flow.

Build an elt pipeline with a Databricks job in workflows to compute product revenue, using tasks like cleanup_retail_datalake, convert_orders, convert_order_items, create table orders, create table order_items, with bronze_base_dir and gold_base_dir.

Run and review a Databricks ELT pipeline job by passing source_base_dir, bronze_base_dir, and gold_base_dir, then inspect task details, notebook outputs, Spark UI metrics, and logs for troubleshooting.

Delete your GCP Databricks workspace and the associated GCS buckets to avoid charges, by signing in to the GCP Databricks console and confirming deletion.