Fundamentals of Data Ingestion with Python

Acquire data from various sources, clean this data, train and evaluate models, and adapt to data that changes over time using high quality, relevant data.

Explore data sources from relational databases and logs to APIs and storage, using Python, and learn data types such as records, mattresses, and graphs, with formats like parquet and ORC.

Build a data pipeline that ingests raw data from logs, performs ETL, validates and enriches records, and stores results in databases or data lakes.

Learn to load csv and xml data in Python using the csv module and pandas, with header handling, type conversion, and chunking for large files.

Load a parquet table with the Apache Aero Library, convert it to a pandas data frame, and examine dtypes and the head to confirm data integrity.

Learn to extract data from unstructured and semi-structured text with regular expressions, parse JSON with Python, and compute metrics like average ride duration for ingestion workflows.

Learn how to use JSON for API data, handle Python type limits, and convert time strings to datetime objects with a custom object_pairs_hook and fixed_time during load and dump.

Learn to make http calls to remote data via apis, using Python and the requests library to fetch json responses, authenticate with a token, and count total trips.

Learn to process event-based data with message queues, subscribe to topics, and decode JSON payloads using Python. Deploy a Nats server in Docker and implement a callback handler for messages.

Find an API first when fetching data from a website or server, and use screen scraping only as a last resort with a parser or regex, APIs are more stable.

Parse HTML with BeautifulSoup to extract the page date and exchange rates, iterating table rows to build a symbol-rate dictionary and return the date and rates.

Use Scrapy to set up a project, write a spider, crawl pages on localhost:8987, and extract date, currency symbols, and ratios as floats, yielding a daily data dictionary.

Automate browser data ingestion with Selenium and WebDriver to load pages, extract data by CSS selectors, and consider API limits, caching, and headless mode for servers.

Understand how a schema acts as metadata for meteorological data, detailing columns like date, snow, Tmax, Tmin, and PG TM. The lecture stresses writing the schema to prevent knowledge loss.

Define and align your system’s entities and their relations from the start to avoid data inconsistencies, using ontologies to standardize definitions like users, buyers, and timestamps.

Define and maintain a comprehensive data schema that documents description, data types, units, constraints, inter-field relations, and plan for schema changes, versioning, validation, and missing data handling.

Explore core database types, including relational databases with sql support, key-value stores like Redis, document stores like MongoDB and Elasticsearch, and graph databases like Neo4j.

Prefer a hosted data platform to ensure production readiness, covering monitoring, security, backup and restore, and scaling, using Google's BigQuery before migrating to a self-hosted solution as data grows.

Explore relational databases with Python using sqlite3, sql queries, and parameters, highlighting safeguards against sql injection, and show how pandas and SQLAlchemy simplify data access.

Explore Redis as a caching layer and JSON serialization for complex values, then see Python code to connect, get, and scan transaction data (SKU and price, 205 transactions).

Explore document oriented databases like Elasticsearch and MongoDB, with schema-less design and no standard query language. Then review graph databases such as Neo4j with Cypher, using Docker and Python demos.

Identify data quality issues and design resilient data pipelines and models to detect and handle errors, avoiding garbage in, garbage out with validation, testing, and pandas tools for missing data.

Find and replace outliers in taxi and apple trading data using pandas, quantiles, and local outlier factor with scikit-learn, illustrating data cleaning and seasonality-aware detection.

Design your data by defining a written schema, understanding where each data piece comes from, and avoiding unused fields to reduce storage, compute costs, and risk of data leakage.

Learn to identify key performance indicators for your data pipeline, focusing on metrics with direct business impact, and set smart, specific, measurable, attainable, relevant, and time-framed KPIs.

Ingest data from diverse sources with Python, validate and clean it, and write a formal schema. Identify more data sources in your organization to improve algorithms with better training data.