The Complete Hands-on Introduction to Airbyte

Airbyte is an open source data integration tool that moves data from APIs, databases, or SaaS to data lakes or warehouses, using connectors and an extensible connector builder.

Discover Airbyte core concepts: source, destination, and connectors; learn how streams and fields organize data, and configure connections with replication options like full overwrite and incremental updates.

Explore Airbyte’s core components, including the config database, web app, API, scaler, discover, workers, and how the UI creates sources and manages sync jobs through the API.

Explore why Airbyte isn't ideal for streaming and batch ingestion latency, and learn to create custom connectors with Airbyte CDK or the connector builder.

Explore Airbyte deployment options, including three paths: open source self-managed community version, self-managed enterprise, and fully managed Airbyte cloud, tailoring features and support to your needs.

Discover how Docker creates isolated containers to run applications with their own dependencies, using Docker files and images, and executing Airbyte with Docker build and Docker run.

Learn to run Airbyte with Docker by installing Docker Desktop, managing containers and resources, starting Airbyte components with Docker Compose, then access the web UI on port 8000.

Take a quick tour of the Airbyte UI, focusing on the connections page with sources and destinations like Airtable and Snowflake. Configure, schedule syncs, and review streams, history, and connectors.

Create and configure an Airbyte connection to replicate data from Google Sheets into Google BigQuery, producing two tables cc and wire with columns date, transaction id, description, and amount.

Create your first Airbyte source with Google Sheets by configuring a service account, enabling the Google Sheets API, and sharing the spreadsheet for access.

Create your first Airbyte destination with BigQuery, replicating data from Google Sheets, attach billing, set project and a customers data set, choose a loading method, and verify the connection.

Make your first sync by replicating data from Google Sheets into BigQuery, then verify the results in BigQuery tables CC and wire via the job history and logs.

Airbyte creates an internal namespace and route tables storing records with airbyte row id, extracted at, loaded at, meta, and data for schema migrations and to recreate the final table.

Airbyte classifies connectors as certified or community. Certified connectors are production-ready and supported by the BI team; community connectors may lack support and SLAs, with open-source code for custom builds.

Discover how Airbyte performs a source-to-destination sync via a sync workflow, using connectors, a queue and workers, with config checks, schema validation, and loading into destinations like BigQuery.

Create a dedicated read-only Postgres user for non-Airbyte databases to enable data transfer, granting read-only access to relevant schemas and tables; the Airbyte Postgres database already has this user.

Explore Airbyte sync modes from full refresh overwrite to incremental append and deduped, using Postgres to BigQuery to illustrate how cursor fields and primary keys prevent duplicates.

Explore how Airbyte detects schema changes before syncing, handles non-breaking and breaking changes, and propagates changes with manual approval.

Discover how change data capture tracks inserts, updates, and deletes in real time and replicates them to databases or other systems using a log-based Airbyte approach.

Configure the Postgres connector to replicate data from Postgres to BigQuery using CDC, including enabling logical replication, creating a replication slot and publication, and switching to CDC in the UI.

Configures Postgres with CDC to replicate data into BigQuery using Airbyte, verifies replication, and explains CDC fields like cdc_lsn and updated_at, including delete handling and CDC limitations.

Learn to build a data pipeline with airbyte that loads fake transactions, runs data quality checks with soda, and produces a customer risk score.

Take your time to learn, watch the video first, then replicate. Focus on tasks, not setup; brief intros to snowflake, airflow, DBT, soda.

generate fake customer transactions with two python scripts: a postgres-backed transaction generator for airbyte and a detection generator that labels fraud and stores results in mysql, triggered by airflow pipeline.

Explore how Apache Airflow acts as a data orchestrator to program, monitor, and schedule workflows, using DAGs, operators, and providers like Airbyte to sync data.

Trigger the data pipeline by running the transaction and detection generators in the Airflow dag. Verify customer transactions and transactions tables in Postgres and MySQL with a fraudulent transaction sample.

Create and configure an AWS S3 bucket and an IAM user, attach a tailored policy, generate access keys, and prepare Airbyte to access the bucket.

Configure an aws s3 destination in airbyte to export postgres data, set access keys, bucket, region, and csv with root-level flattening; test and finalize setup.

Create a Postgres to S3 connection in Airbyte by selecting the Postgres source and S3 destination, enabling the customer transactions stream, and setting the manual schedule.

Create an Airbyte connection from MySQL to S3 to transfer the labeled transactions table; name it load labeled transactions and set the schedule to manual.

Learn the write audit publish pattern that stages data, audits it for quality with a framework like soda, and only then moves valid records to production for downstream consumers.

Create the raw to staging connection in airbyte from the S3 source to Snowflake, using the staging schema and enabling streams for customer transactions and label transactions.

Learn to validate staged data with soda, a data quality framework, by defining checks in a YAML file with soda core and soda SQL to ensure tables are not empty.

Learn to implement data quality checks with Soda for Snowflake, including not empty tables, duplicate IDs, schema validation, and cross-table reference checks between customer and labeled transactions.

Publish the customer metrics table with dbt in snowflake, calculating a risk score from fraudulent transactions. Transfer staging data to production via dbt models for customer transactions and labeled transactions.

PyAirbyte lets engineers code data ingestion, speeding prototyping and integration with airflow and link chain. Avoids docker, kubernetes, or cloud access, and offers a code-based option over the user interface.

Explore pyairbyte, an open source python library that ingests data from systems using airbyte connectors, enabling Genie, machine learning, data warehousing, and analytics with duckdb sql and pandas integration.