Fivertran ETL from Scratch

Connect a Postgres instance via pgadmin, sync the countries, departments, and employees tables, and migrate the entire schema and data from Postgres into a Redshift data warehouse.

Configure a PostgreSQL source connector with host, port, user, TLS, and optional SSH tunnel; enable logical replication using test decoding or page output, and set changes to allow all.

Configure a Redshift destination in Fivetran, set host and port, and map databases Patron and Whiterun. Choose direct or SSH tunnel connections, whitelist AWS US IP, then save and test.

Sync data from Postgres to a Redshift warehouse using Fivetran, perform the initial sync, and create the PostgreSQL Artist Public schema with countries, departments, employees, and an audit metadata table.

Validate Syncdb data by inspecting the employees table and its column data types and schema. Note the new Fitton ID and Fitton Audit metadata, including the sync date and timestamp.

Resync the employees table from the source by selecting a schema and auditing the changes, capturing update IDs as a unique combination in the audit table for the target.

Understand Python metadata in Fivetran ETL, including per-table columns Patron ID, Frighten deleted, and Frighten sync timestamps, plus the Filter Audit table for sync object audits.

Manage source connector properties, view and filter logs, manage schema and three tables (countries, departments, employees), and configure sync, notifications, and pause or delete actions.

Sync data from an aws s3 bucket to a redshift data warehouse using the s3 connector, configuring external id, iam policy, and role for the locations.csv file under destination schema.

Syncs s3 data to redshift via an etl process, creating schema underscore s3 and tables locations, fighter, and audit, with metadata columns and a successful data load.

Insert a new employee record in the source, then sync it to the target and verify via logs and an audit table that only the new record was synchronized.

Delete Argentina and Australia entries from the countries source and sync again to observe fivetran marking deletions with a deleted flag and audit history in the target.

Learn to design transformations with sql statements or Dvb-t, trigger transformations on data changes or schedule them, and integrate Fivetran with DB2 core to run DB2 models.

Perform a sql data transformation that concatenates first name and last name with a space and retrieves salary to create employees_details, with scheduling by data refresh or time trigger.

Create transformation joining employees and departments, select columns, and create a view employees_view with full name and 10% salary increase; lowercase emails and append mail.com, then execute and test.