Data Engineering Fundamentals with Prefect Workflow

Learn how data engineering serves as a precursor to data analytics and data science by building data infrastructure on the cloud using ETL and data pipelines.

Understand how machine learning, data science, and artificial intelligence relate, illustrated by autonomous cars. See supervised learning with labeled road signs that builds a model to drive and brake.

Compare ETL and data pipelines, from legacy structured data and data warehouses to modern cloud pipelines built from modular tasks with branching and DAG concepts.

Compare airflow and prefect as task orchestration tools for data pipelines, highlighting dag versus flows and the machine learning workflow from data collection to deployment and monitoring.

Explore virtualization in cloud computing by creating and accessing Oracle Linux virtual instances through secure connections, IP addresses, and public-private keys.

Learn to create an Oracle Linux virtual instance on Oracle Cloud, secure it with public-private keys, and connect via SSH using Putty, with steps for compartments, VCN, and ports.

Download putty, open puttygen to generate RSA public and private keys, save the private key locally, and copy the public key for cloud instance access.

Create a compartment if missing, then provision a virtual cloud network (VCN) in OCI with internet connectivity, review security rules, and prepare for the Oracle Linux instance.

Create and configure a Linux 9 virtual machine on Oracle Cloud Infrastructure using the free tier, upload your public key, and access it via PuTTY.

Connect to a virtual machine via putty using the IP, private key, and SSH port 22; log in and explore Linux basics for workflow automation and Python 3.x usage.

Provision a Linux VM GUI on Oracle Cloud by following an 18-step process, installing the server with GUI and TigerVNC, configuring a VNC password, and running a GUI setup script.

Set up a Linux server gui and install TigerVNC, then configure the vnc service, firewall, and port 5903. Connect via a vnc client to access the Prefect workflow gui.

Explore the Prefect cloud environment for task orchestration with flows and Python-defined tasks, using a cloud dashboard, API login, and virtual machine workflow.

Install prefect on a linux vm with gui, verify python compatibility, run pip install prefect, check the 2.14 api version, then connect to prefect cloud to run tasks and flows.

Install the Prefect client on an Oracle Cloud VM, connect to the Prefect cloud dashboard for your data pipeline, authenticate with an API key, and schedule your first flow.

Experiment with a simple Python hello world flow in Prefect IO, deploy it on a client machine, and monitor its minute-by-minute execution and dashboard status via Prefect UI.

Explore Prefect workflow documentation to learn core concepts like flows and tasks, and master deployment and scheduling via cron jobs, intervals, and dashboards.

Authenticate to Prefect Cloud, run a simple flow with decorated flow and task, and observe flow-first execution and task dependencies across extraction and load steps.

Provision autonomous databases in the Oracle Cloud that automate patching, upgrades, and tuning. Explore OLTP, data warehousing, Oracle Apex, and JSON data workloads with self-managing, self-securing, self-repairing infrastructure.

Learn how compartments define a boundary in Oracle Cloud to isolate resources such as databases, VMs, and VCNs, and how to create or delete compartments with the removal rule.

Provision an Oracle autonomous database on the free tier for transaction processing or data warehouse, configure workload settings, and explore built-in ODI for data integration and ELT.

Connect to the Oracle Autonomous Database in three ways. Use the oci dashboard web sql client, the Oracle sql client with sql developer, or Oracle Apex from the dashboard.

Connect to the Oracle Autonomous Database via the web edition of SQL Developer, explore the always free tier options, and run sample queries like selecting all from the customers table.

Learn to connect Python to an Oracle Autonomous Database with the Oracle DB library. Create, insert, and retrieve data using a sample script on Oracle Cloud free tier.

Learn how to retrieve data from an Oracle Autonomous Database using a Python script, including connection setup, cursor usage, and executing a select query to fetch table data.

Compare OLTP and OLAP, and see how data pipelines and ETL support building a data warehouse through data engineering. Use Prefect to orchestrate tasks and reveal dependencies.

Learn to build a two-task Prefect data pipeline with dependencies, using Python to interact with an Oracle database, including access control configuration for Linux and local IPs.

Compare webhooks, mqtt, and web sockets by communication style differences. Webhooks push events to a URL; mqtt uses a broker for publish-subscribe in IoT; web sockets enable bidirectional communication.

Explore event driven workflows with Prefect and GitHub through hands-on webhooks, detailing manual setup of a Prefect event webhook and a GitHub webhook, and previewing automated triggers on issues.

Automate GitHub issue events by deploying a webhook-driven workflow with Prefect, using a programmatic deployment, a work pool, and JSON parameters to trigger automated events.

Explore the data engineer career path, from entry level to roles like data analyst and data architect, and learn core skills in SQL, NoSQL, Python, Java, and data pipelines.