Data Engineering with Google Datafusion and Big Query (CDAP)

Set up Google Cloud Storage buckets to manage semi-structured data for the data fusion project. Define input, processing, and output folders to orchestrate file movement with data fusion and Dataproc.

Set up our data fusion environment, build a serverless BigQuery data warehouse in a data lake format for bronze, silver, and gold datasets, and enable SQL-based analytics.

Explore the structure of semi-structured JSON with nested arrays and dictionaries and learn how Data Fusion normalizes it with a few clicks, no code required.

Upload credit.json to the cloud storage bucket and connect Data Fusion Studio as the pipeline source. Use Wrangler to parse JSON and explode arrays into columns for downstream processing.

Finish the data fusion pipeline and use preview mode to debug with GCS input, Wrangler transform, and trash sink, then inspect logs to diagnose schema errors.

Learn to configure a BigQuery sink in data fusion, connect to the bronze dataset, and create a test table while previewing the pipeline to validate data before deployment.

Troubleshoot and resolve a Wrangler import bug when reusing a json pipeline across data fusion instances, recreate Wrangler, copy recipes, reconnect, and reapply transformations.

Branch the pipeline to split cards and clients data into two tables, propagate the input schema, and join via the id key to save cards and clients in BigQuery.

Move files from input to processing and then to output using Google Data Fusion, with regex filtering and isolated stages to ensure no reprocessing and accurate writes to BigQuery.

Build a new pipeline that reads from BigQuery as the source, moving data from bronze to silver with partition date filters and data preview.

Wrangler enables low-code data transformations for the credit server pipeline in BigQuery. Transform bronze to silver, join name and surname, and parse addresses with regex while propagating output schemas.

Explore Wrangler's advanced functions in data fusion to parse, cast, format, and calculate data, then build a BigQuery pipeline with filters, data quality checks, and a temp table.

Transform data with BigQuery SQL inside a data fusion pipeline and later run queries from within data fusion to orchestrate BigQuery workflows, including time partitioning considerations and the pseudo column.

Demonstrates executing a Datafusion query with a post-action to BigQuery, rollback of date formatting, and using temporary tables to isolate steps, then deploys and validates results.

master ingestion-time partitioning in bigquery to speed queries on bronze data, save costs, and boost pipeline reliability by date-partition filters and merge upserts for the silver table.

Master merging data into BigQuery with a merge into statement, using distinct preparation and a source-to-target upsert pattern to keep the pipeline idempotent in Data Fusion.

Delete temp tables and use parameterized macros in a Google Data Fusion and BigQuery workflow. The lesson covers bronze pipeline execution, input upload simulation, and partitioned reads for efficiency.

Configure dag-based dependencies between bronze and credit pipelines with inbound triggers and success-based handoffs. Set up cron schedules and manage max concurrent runs to automate end to end data workflows.

Diagnose and resolve the disk total GB quota exceeded error in Google Data Fusion pipelines by increasing the compute engine persistent disk quota in the target region and reviewing logs.

Receive the schema and merge query and build the cards silver pipeline by creating a partitioned BigQuery cards table and wiring it through Data Fusion with the bronze pipeline.