Master Data Engineering: Concepts to Production

Explore how SQL, the standard language for relational databases, enables efficient querying, data extraction, and maintenance of tables like doctor and patients, linked by IDs.

Set up MySQL development environment by installing MySQL on Unix, Windows, macOS using yum, apt, or brew; start the service and log into MySQL to run queries with MySQL Workbench.

Create the Cal Tix database and build doctors and patients tables, defining primary and foreign keys, while performing basic SQL tasks in MySQL workbench.

Execute insert statements to populate doctor and patient tables, enforce not null constraints on doctor names, and ensure foreign key references match existing doctor ids for successful data insertion.

Extract data from the doctor and patient tables using select and where, filtering patients by doctor id and listing doctors in the cardiology department.

Learn to count records and group by doctor IDs to determine how many patients are assigned to each doctor, using count(*) and group by.

Rank doctors by patient totals using an order by clause with a count alias, and switch from ascending to descending to see doctors ranked from highest to lowest.

List doctors with a patient count of five or more by applying the having clause after group by and ordering by count. Explain why where cannot filter aggregate columns.

Learn how to join patients and doctors using an inner join, with aliases and explicit column selection. Explore join types, unions, ctes, and subqueries to query multiple tables.

Learn how a left join returns all rows from the left table and matched rows from the right table, with nulls where there is no match. Compare this to the right join using doctor and patient tables.

Explains full outer join by uniting left and right joins, filling nulls for unmatched rows, and uses union in MySQL to simulate it, then contrasts union and union all.

Learn how common table expressions (CTEs) simplify complex queries, using a doctor and patient table to find patients over 12 still tagged to a pediatrician with date_sub.

Learn how subqueries function as inline queries and compare them with reusable common table expressions. See an example using a CTE and a subquery to yield the same results.

Learn to create and alter tables, add and modify columns, and update data in the Celtics database, including a visits table, temporary tables, and when to truncate or drop tables.

Learn how to use date and string functions in MySQL, including date_add, subdate, date_diff, and string operations like concat, concat_ws, substr, length, with pattern matching via like and regex.

Explore window functions, including aggregate, rank, and value types, and see how partitioning by department and ordering by sales amount yields top performers and department totals.

explores window functions with a real-world example to find the top-earning doctor per day by joining doctor and visit tables, using a cte to rank by date and total fees.

Explore rank, dense rank, row number, and the average window function to compare doctor earnings against daily clinic totals and understand tie effects.

Explore database design principles, including normalization and denormalization, to create efficient schemas that minimize data redundancy and enforce transaction control.

Explore first normal form by eliminating repeating groups in an order table, splitting into multiple rows so each cell holds a single value for order id, product, and quantity.

Learn how second normal form eliminates partial dependencies by moving price to a product price table, so non-key price depends on product rather than the order.

implement the third normal form by removing transitive dependencies and moving customer data to a separate order-customer mapping table, splitting the order table into order details and customer mapping.

Denormalization duplicates data to speed queries, as in orders sharing the same customer information. It trades data consistency for faster access, useful for real-time analytics or caching.

Normalize data to remove duplicates, improving consistency and storage efficiency. Select appropriate data types, use indexes judiciously, and design for growth with partitioning or sharding to ensure scalability and performance.

Learn how atomicity, consistency, isolation, and durability make database transactions all or nothing, follow rules, stay isolated, and remain permanently stored for reliable data management.

Master SQL performance tuning by creating and managing indexes and analyzing queries. Compare subqueries and CTEs to optimize execution plans and boost query speed.

Master transaction control by using commit to finalize changes, rollback to undo changes, and a save point to restore a saved state in a shopping cart workflow.

Discover how ETL extracts data from multiple sources, transforms and cleanses it with SQL, and loads it into a centralized data warehouse to enable accurate reporting and data-driven decisions.

Explore how data pipelines extract, transform, and load data from sources like CRM, weblogs, and APIs into targets such as Hive, AWS S3, and Tableau dashboards.

cleanse and transform data with sql by validating names in the patient table using regex, fixing entries like d0n to don via a cte-driven update.

Explore data warehousing concepts, including fact, dimension, and snapshot tables, and compare star and snowflake schemas. Learn how a time-variant, non-volatile repository integrates data from multiple sources for fast analysis.

Explore how fact tables store measurable business data, how dimension tables provide descriptive context, and how snapshot tables capture data at a specific moment in the data warehouse.

Compare star and snowflake schemas with a central fact table, surrounding dimension tables, and a snowflake extension that creates transitive dependencies through category and hierarchy.

Analyze slowly changing dimension tables and their three types: type 1 updates the row, type 2 adds history with dates, and type 3 tracks previous values.

Define goals, design architecture, and plan implementation for a data warehouse, selecting star, snowflake, or hybrid models, and addressing ETL, data quality, governance, security, and scalability.

Design an end-to-end ETL data pipeline that extracts from CRM, ERP, and web analytics sources into a data warehouse. Apply cleansing, transformations, and load data into dimension and fact tables.

Explore what an operating system is, what Unix is, its history, and Unix vs Linux, plus its role in data engineering and modern data platforms.

Explore how an operating system manages hardware, memory, storage, and program execution, provides a user interface, and enforces permissions to keep the system secure.

Explore Unix, a foundational and secure operating system from the 1960s–70s, and its influence on Mac OS, Linux, servers, and data tools, enabling transferable skills across technology and data work.

Trace Unix history from late 1960s Bell Labs leaders Ken Thompson and Dennis Ritchie. Observe how C made it portable, inspiring BSD, Solaris, Linux, and Mac OS.

Clarify the difference between Unix and Linux: Unix is the original, proprietary system from the 1960s; Linux is a free, open-source, Unix-like alternative used in servers and cloud.

Embrace unix and linux environments to master data engineering, use the cli for file management and rapid data cleaning with grep and awk on Hadoop, Spark, Kafka, and Airflow pipelines.

Explore the four layers of Unix architecture—hardware, kernel, shell, and application software—and learn how the shell interfaces with the kernel to run programs.

Navigate the unix file system and manage files using cd and pwd, then view details with ls -l and sort by time with ls -lt, reversing order with r flag.

Explore essential unix file and directory management, including mkdir, cd, pwd, ls, touch, rm, and copy and move operations for data engineering workflows.

Master three Unix file creation methods: touch, cat with redirection, and vi editor, and learn to write, view, and save content with proper commands.

Use cat to print full file contents, more to fit the screen, and head or tail to view the first or last lines for debugging logs.

Explore file permissions and ownership, including read, write, and execute. Learn to change permissions and ownership using commands like own and group, and inspect files with ls -lrt.

Explore how user, group, and others gain read, write, and execute permissions, and use chmod to add, remove, or set these rights on files.

Explain octal mode for updating file permissions by mapping read, write, and execute to digits. Show hands-on chmod examples like 755 and 754 that assign owner, group, and others permissions.

Learn how c-h and change group commands adjust file ownership and groups, including Linda as owner and admin group as owning group.

Explore text processing tools using grep, said, and awk to search patterns in logs, identify exceptions, and pinpoint root causes in spark job failures.

Explore the stream editor for parsing and transforming text, including global replacements like replacing info with debug in files or Spark logs, a key data engineering tool.

Orc enables data engineers to extract text by pattern, such as lines with the error keyword. Pipe the output to grep to filter for exception.

Explore how a unix process is created and managed by the kernel. Track its life cycle from created to ready, running, blocked, and terminated, using ps, top, and kill.

Learn how the ps command displays active processes, lists all processes with ps -e, uses head -5 for top items, and top for real-time cpu usage and memory usage monitoring.

Explore data compression and archiving with gzip and tar, and distinguish archival from compression, noting that compression reduces size while archiving groups files into a single folder.

Learn how to securely transfer files using SCP, securely log in with SSH, and access remote storage with a network file system (NFS), including mounting remote paths.

Master regular expressions for matching and processing in engineering, using grep to extract email IDs from emails.txt that start with lowercase letters, have @, and end with .com or .org.

Explore Unix shell scripting and learn how shell scripts automate repetitive data engineering tasks by executing commands with bash or zsh.

Write and run your first shell script with vi, saving hello world in scripts directory. Compare ssh invocation and dot slash execution; learn to use variables with chmod.

Implement shell scripting techniques to explore control structures, using for and while loops to determine even and odd numbers from 1 to 20, with readable indentation.

Explore functions and modular scripting in Unix by defining a square calculator function that uses the first argument, demonstrates reusable blocks, and prints the square of a given number.

Master redirecting stdout and stderr to log files for debugging in data engineering; create messages.sh, provoke errors from a missing file, and verify stdout.log and stderr.log contents.

Learn how to work with files in shell scripts by writing and appending to myfile.txt, checking the return code, and displaying file contents.

Learn to handle errors in shell scripts by debugging a for loop, fixing syntax issues, and redirecting stdout and stderr to log files to verify correct execution.

Use unix to build a data pipeline that reads employee details csv, validates numeric experience, runs while loop, adds salary header, computes salary as experience times 1000, and writes csv.

Master cron job scheduling with crontab using five fields for minutes, hours, day, month, and weekday. See a hands-on example executing a hello world script and logging to cron output.txt.

Implement unix-based best practices for data engineering, including version control with git, modular scripts, cron automation, graceful error handling, log redirection, nohup, and minimal permissions using oc, z, and grip.

Explore the history, features, and architecture of Python and its essential role for data engineers, from readability to data science and data workflows.

Explore Python's intuitive syntax and interpreted, line-by-line execution. Harness its cross-platform reach, extensive standard library, dynamic typing, multiple paradigms, vast ecosystems, and strong community for readable and maintainable code.

Explore how Python code runs from parsing source into an abstract syntax tree to bytecode execution by the Python virtual machine, and compare interpreters with compilers.

Python powers data engineering with simple, readable code that maintains complex data pipelines, using Pandas, NumPy, and Spark or Dask for scalable ETL and data source integration.

Download and install python across Windows, Unix, and macOS using official pages, apt-get, or brew, then verify with which python3. Explore using an integrated development environment like PyCharm or IntelliJ.

Install IntelliJ IDEA community edition on macOS, set up a Python project with a virtual environment and Python 3.12, and use Copilot for debugging, code completion, and data engineering productivity.

Set up a Python project with packages source and resources, create a main script Celtics_main. Explain the if __name__ == '__main__' pattern, indentation, and printing 'Hello data engineers'.

Set up the project structure with resources and source directories, visualize the sales daily transaction exports csv using a csv plugin, then read and process the data.

Master Python basics, including syntax, variables, data types, operations, and expressions, and learn to define and call functions, pass arguments, return values, import modules, and read files.

Learn to define Python variables with dynamic typing. Practice adding A and B, and concatenate outputs by converting integers to strings.

Learn how Python methods modularize code by defining functions with parameters, calling them to sum numbers, and returning values to the caller.

Learn to read a csv using the read file method in the file utils module of the process files package, skip the header with an iterator, and print each row.

Master data engineers learn Python control flow using for and while loops, lists, tuples, and sets, plus if, elif, and else to categorize sales data from csv files.

Explore Python lists, tuples, and sets by converting a csv reader to a list, iterating through items, and using built-in methods like append, pop, and index.

Explore Python list slicing to create a new list by extracting portions using start, stop, and step, with positive and negative indices and inclusive/exclusive behavior.

Identify and extract customer names from a list of orders using a for loop and the enumerate function, filtering values greater than 25, and build a valued customer list.

Learn to use if, elif, and else in Python to categorize customers by order value. Apply and or logical operators and proper indentation to create clear decision blocks.

Organize your Python project with a utils folder and a categorize customers function that returns valued, medium, and least valued lists, then import and use it in main.

Learn how to use the while loop in Python to process an orders list, cast order values to float, categorize customers, and decide when to prefer for loops.

Explore Python tuples as immutable data structures, distinguish them from lists by round brackets and indexing, and learn when to use tuples for fixed-size data while considering memory use.

Explore Python sets and contrast them with lists, highlighting that sets do not allow duplicates and are unordered, demonstrated through a code example.

Learn to map customer names to order values using Python dictionaries, with a get names and order value function and enumerate to pair indices with values in ETL workflows.

Master data engineering by writing JSON files for ETL loads, ensuring readable code with docstrings and comments, and using lambda expressions to shorten method definitions.

Open the file in write mode and dump the customer dictionary to a json file using python's json library with four-space indentation, demonstrating etl by loading csv data into json.

Document Python code with docstrings in triple quotes to improve clarity, readability, and collaboration across modules, classes, and functions, following best practices for maintainable, start-of-module and function documentation.

Wrap Python file reads in a try-except block to gracefully handle file not found errors, returning a blank list and printing a clear message, and learn common built-in exceptions.

Explore Python lambda expressions for quick one-line functions. Learn the syntax and how to apply them with map and filter to transform and select list elements.

Build a Python etl pipeline that extracts doctor visit data from a MySQL database, transforms it to count daily visits per doctor, and loads the results into a json file.

Explore the big data landscape, driven by volume, variety, and velocity, and see how Hadoop manages large data with HDFS and MapReduce while Spark enables real-time processing.

Explore big data fundamentals, including the five v's—volume, velocity, variety, veracity, value—and their use cases and challenges for data engineers.

Big data enables real-time analytics, predictive modeling, and personalized insights across healthcare, finance, retail, and transportation, powering fraud detection, risk management, and optimized operations.

Explore the key challenges in big data processing—volume, variety, velocity, veracity, security, and scalability—and how tools like Hadoop HDFS, Apache Kafka, and Spark Streaming address them.

Explore the big data technologies ecosystem and learn why organizations choose big data over traditional data processing systems.

Explore how data sources, data storage, data processing, ETL, analytics, visualization, data management, and data security form the big data ecosystem, with cloud platforms enabling scalable development and deployment.

Start Hadoop and Yarn daemons, then use the HDFS dfs command to list files and directories, create directories, and delete them recursively in HDFS.

Understand how traditional data processing struggles with huge data volumes and batch insights, while big data processing handles real-time streams and any data type, with horizontal scaling.

Learn to manage files in HDFS by creating a data directory, uploading with put or copy from local, downloading with get, copying directories, and deleting files with verification.

Learn to view HDFS file contents with cat, head, and more; verify data integrity using hdfs dfs -checksum; and inspect statistics with hdfs dfs -stat in a pseudo distributed cluster.

Discover Apache Hadoop and its core components, including HDFS, MapReduce, Yarn, and common utilities. Analyze HDFS architecture, the MapReduce framework, how MapReduce jobs execute, and Yarn architecture.

Explore Apache Hadoop, a distributed storage and processing framework with HDFS, MapReduce, Yarn, and Hadoop common, delivering scalable, fault-tolerant, cost-effective, flexible data handling for warehousing, analytics, and machine learning.

Explore the four key Hadoop components—HDFS, MapReduce, Yarn, and common utilities—and learn how HDFS stores large data sets with high throughput, while MapReduce and Yarn coordinate processing and resources.

Explore the HDFS architecture with a primary name node, secondary name node, and data nodes storing 128 mb blocks replicated across nodes for client access and centralized metadata management.

Explore MapReduce in Hadoop, learning how input files split into chunks, map to key-value pairs, shuffle and sort, then reduce to word counts through parallel processing for big data.

Explore yarn, the yet another resource negotiator in the Hadoop ecosystem, and learn how resource manager, application master, and node managers coordinate containers across a cluster.

Explore the five Hadoop or Spark setups: standalone, pseudo distributed, fully distributed, cloud based, and Hadoop as a service, and identify ideal use cases.

Install OpenJDK 8 to run Hadoop, since Hadoop works best on Java 1.8. Configure JAVA_HOME and update PATH in .bash_profile, then verify with java -version.

Install and configure Hadoop locally, set fs.defaultFS to localhost:9000, enable yarn, format the name node, start daemons, and verify via hdfs and the NameNode and ResourceManager web UIs.

Explore Hive, a data warehousing solution with a sql-like language on Hadoop to query large datasets in HDFS or S3, featuring Hive Server 2, Metastore, ACID, and replication.

Hive uses schema on read for unstructured data and HQL batch processing for large-scale OLAP analytics on Hadoop, while traditional RDBMS enforces schema on write and SQL for OLTP.

Explore how Hive analyzes e-commerce clickstream data, processes electronic health records to spot care trends, detects fraud through real-time financial transactions, and monitors telecom network logs to optimize performance.

Explore Hive architecture, including the UI, driver, compiler, meta store, and execution engine, and how Hive uses MapReduce tasks with deserializer and serializer in Hadoop.

Explore Hive query language (hql) to query and analyze data stored in Hadoop, translating sql-like queries into MapReduce jobs for batch processing, with partitioning and bucketing to optimize performance.

Install Hive on a local machine after Hadoop, configure hive-site.xml and core-site.xml, set Hive home and path, and initialize a Metastore in MySQL using schema tool.

Master Hive tables, including managed and external types, and learn how partitioning, bucketing, and the Hive metastore relate to data locations in HDFS or remote storage.

Create a hive table with external or managed options, define columns with comments, and configure partitioning, bucketing, row format, and file format for efficient storage and retrieval.

Explore Hive file formats, including text files, sequence file, RC file, Avro, ORC, and Parquet, and learn how to choose the right format for schema evolution, compression, and analytical workloads.

Create a managed Hive table named employee with id, name, and salary; load data from a local file, then query using where, order by, group by, and a join condition.

Load data from a local file system into a hive table using load data local inpath, then verify the five records were inserted into the table.

Learn to query a hive table with where, sort by descending salary, aggregate by salary counts, and join tables (inner and left joins) using real-world employee and manager data.

Explore how Hive user defined functions enable custom logic on top of tables to query and transform data, with built-in and custom UDFs and the difference from aggregate functions.

Explore the core of Apache Spark, its ecosystem and architecture, and compare Hadoop and Spark, as part of mastering data engineering.

Explore Apache Spark, a fast in-memory engine for batch, stream, and interactive queries, with Java, Scala, Python, and R APIs, enabling real-time fraud detection from CRM data.

Explore the Spark ecosystem, from Spark core and cluster management to libraries like Spark SQL, graph processing, MLlib, and Spark Streaming, with storage options and language support.

Explore Apache Spark architecture by examining the driver program, cluster manager, and executors, and learn how their roles enable scalable, fault-tolerant data processing for real time streaming and batch workloads.

Compare Hadoop MapReduce and Spark for big data processing, highlighting performance and fault tolerance. Spark's in-memory processing enables faster analytics and real-time insights.

Install spark by downloading a compatible version for Hadoop, configure spark home and path, verify with spark-submit, then run a quick pi spark hello world in the pi spark shell.

Submit a spark job to trigger the driver to build a dag, split it into stages, and assign tasks to executors across the cluster, reflecting wide transformations and shuffles.

Discover how Spark session unifies Spark context, sql context, and Hive context to run data frames, rdds, and sql queries via Spark shell, PySpark, Spark sql, and the cluster manager.

Explore Spark data frames as named-column data sets and compare datasets to RDDs, highlighting fault tolerance through lineage, distributed partitions, and parallel processing.

Navigate the Spark UI to debug applications by inspecting the context web UI and its tabs: jobs, stages, storage, environment, and executors; monitor status and executor health.

Learn spark coding in scala, pyspark, java, and r; practice scala by creating a data frame from records, naming columns, and displaying it with df.show, using spark context and session.

Learn to start PySpark, create a data frame from a list of tuples with spark.createDataFrame, specify headers, view with df.show, and inspect a Spark job in the web UI.

Set up a PySpark project in IntelliJ IDEA community edition, install PySpark, initialize a SparkSession, and use data frames and Spark SQL to filter and query data.

Learn how continuous integration and continuous deployment streamline data engineering workflows, automatically integrating code changes, testing integrity, and deploying updates to production or staging with rollback capabilities.

Enable rapid, reliable data pipelines by implementing ci/cd, automated testing, and collaborative deployment across environments, reducing downtime and human error.

Develop, validate, and deploy data pipelines through a five-stage ci/cd workflow—development, ci validation, cd deployment to staging and production, and production go-live—featuring version control, code reviews, tests, and compliance checks.

Git is a distributed version control system that tracks file changes and enables collaboration through branching, history, and merges. It supports offline work and syncing with remote repositories like GitHub.

Sign up on GitHub, create a private repository named zozo, add a readme and gitignore, select the Apache license, set main as default branch, and use pull requests.

Clone a git repository from remote to local by installing git, creating a local directory, and running git clone with the remote URL, then handle private versus public access.

Learn to create a feature branch from main, commit and push changes, and merge via pull request to production, using push, pull, and merge in git.

Learn CI/CD tooling for data engineers, with GitHub, GitLab, Bitbucket, Jenkins, CircleCI, and GitHub Actions. Cover deployment and testing with Kubernetes, Docker, Terraform, PyTest, and Great Expectations.

See how CI/CD automates testing and deployment to speed up data pipeline releases and improve reliability across environments. Understand challenges like data dependencies, test data availability, and cross-environment consistency.

Explore data quality and governance by identifying key dimensions—accuracy, completeness, consistency, timeliness, and reliability—and see how data issues in customer records affect ETL processes and business decisions.

Learn the key data quality dimensions: accuracy, consistency, uniqueness, timeliness, validity, and completeness, and how to ensure data stays true, non-contradictory, up-to-date, and complete.

Define data quality metrics with KPIs for accuracy, error rate, and uniqueness. Measure data accuracy rate, duplicate record rate, and distinct value count to gauge overall data quality.

Learn data profiling to understand data structure, content, and quality, its purpose, and the techniques and tools to reveal missing values, errors, and cardinality for cleansing or analysis.

Learn how data cleansing corrects inaccurate, incomplete, or inconsistent data, and data standardization brings diverse formats, such as JSON and XML, into a single CSV, improving reliability for decision making.

Explore data cleansing and standardization techniques, including formal standardization of dates, addresses, and data type conversion, and apply cleansing methods for missing values, duplicates, error correction, outliers, and data validation.

Define data governance as the framework of authority, controls, standards, and processes that treat data as a valuable asset. Explain the data lifecycle, access controls, and risks of breaches.

Apply data governance principles across stewardship, ownership, quality, and metadata management, then ensure security, privacy, standards, master data management, and architecture for compliant integration and risk.

Explore centralized, decentralized, hybrid, command and control, and collaborative data governance models, and compare their pros and cons for consistent policies, agile responsiveness, and accountability.

Explore data stewardship, its definition and importance, and the roles of data stewards, including data quality management, governance, metadata management, access and security, and lifecycle management.

Master metadata management explains metadata definitions and the three types—operational, business, and technical—and shows how to create, store, and govern metadata for better data discovery, lineage, and understanding.

Explore data lineage, its significance, and techniques for tracking data through pipelines. Use automated discovery, data profiling, and SQL queries to trace root causes of data quality issues.

Explore the essentials of data privacy laws, including GDPR and HIPAA, and how they govern the collection, use, storage, and sharing of personal information for data engineers.

Explore how data engineers enforce governance and data quality through policies and metadata management. Implement security, compliance, profiling, cleansing, monitoring, and lineage tracking and versioning for auditing.

Explore the four cloud architectures—public, private, hybrid, and multi-cloud—and learn how pay-as-you-go public resources, private cloud data security, and workloads moving between public and private clouds shape modern deployments.

Explore cloud computing fundamentals, models, and offerings, and learn how scalable, elastic, cost-effective resources provide compute, storage, database, and networking accessible from anywhere, with strong security.

Explore the three cloud models: IaaS, PaaS, and SaaS, highlighting how IaaS rents hardware, PaaS adds platforms and tools, and SaaS delivers ready software over the web.

Explore cloud offerings across compute, storage, databases, networking, and analytics, with AWS examples such as EC2, S3, Glacier, RDS, Aurora, VPC, SageMaker, and Bedrock, plus CI/CD and IoT features.

Explore AWS global infrastructure, regions, and availability zones for low-latency data engineering architectures. Understand pay-as-you-go pricing, free tiers, and hands-on use with the AWS Console and CLI.

Create an aws account, verify your email, and enter credit card details to access the aws management console; set a zero spend budget in billing with email alerts.

Explore AWS EC2 and AWS Lambda to provision resizable compute capacity and run code on demand, comparing virtual servers with serverless execution triggered by data changes or user events.

Create an EC2 instance in us east 1 using a free tier linux t2.micro, generate a key pair, connect via SSH or EC2 connect, and verify by creating a file.

Create a function in the aws lambda console and choose the python runtime. Deploy, then test with a new test event and review logs to confirm a 200 status.

Explore AWS S3 and EBS storage options. S3 provides scalable, durable object storage in buckets for backups, archives, and data lakes, while EBS offers low-latency block storage for EC2.

Create an S3 bucket, then upload files from the console or from EC2 using the AWS S3 CLI, and configure IAM credentials for reliable S3 access.

Explore elastic block store (EBS) and examine a volume attached to an EC2 instance, then learn to create and attach a new volume within the same region.

Explore two popular AWS databases, RDS and DynamoDB, and learn how to choose relational versus NoSQL designs for scalable, high-performance, serverless applications.

Create an AWS RDS MySQL database using the free tier and single availability zone, note endpoint URL, install MySQL client on the EC2 instance, then connect and run show databases.

Explore aws networking with vpc, a private isolated cloud network you control, including subnets, route tables, gateways, security groups, and route 53 for scalable dns, health checks, and traffic routing.

Learn how AWS IAM manages access and identity, including users, groups and roles, and how AWS Secrets Manager securely stores passwords, API keys, and database credentials.

Create IAM policies for EC2 and S3, assemble them into a user group, and add users to grant access; assign a role enabling EC2 to access S3.

Create a secret in AWS Secrets Manager and retrieve it with Boto3 in Python, then fix IAM permissions to access Secrets Manager securely.

Explore aws monitoring with CloudWatch to track metrics, logs, and events, set alarms, and trigger SNS notifications for scaling; learn CloudFormation to model and deploy resources via templates.

Master data modeling and architecture to organize and structure data for quick, informed decisions, ensuring scalable, reliable data flows across tools and systems.

Explore data modeling concepts with a blueprint of data, detailing conceptual, logical, and physical models, entities, relationships, attributes, data types, tables, indexes, and methodologies.

Explore six data modeling methodologies: entity relationship modeling, object oriented, dimensional, data vault, agile, and NoSQL, and how each supports relational design, warehousing, and agile data evolution.

Introduction to data architecture explains how data flows through a system and serves as the blueprint for storage, access, and secure use, contrasting architecture with data models.

Identify the six components of data architecture—data sources, storage, integration, governance, processing frameworks, and data access—and illustrate tools like ETL, data lakes, Collibra, Hadoop, Spark, and APIs.

Explore entity relationship modeling to visualize entities, attributes, and relationships. Learn about er diagrams and their components, with examples like a customer placing orders in a one-to-many relationship.

Explore normalization and denormalization in data design. Learn how breaking data into smaller tables reduces redundancy and how denormalization speeds queries by reducing joins, from 1NF to BCNF via examples.

First normal form ensures each column holds a single value and each record is unique by removing repeating groups, simplifying library borrowing lookups.

Learn how second normal form eliminates partial dependencies after achieving one NF, by splitting data into book, borrower, and book-borrower tables to remove redundancy.

Identify the criteria for third normal form, requiring two nf and no transitive dependencies between non-key attributes and the primary key, as shown by separating author and library data.

Explain Boyce-Codd normal form: every determinant is a candidate key, author determines genre. This violates BCNF, prompting a book branch table and final tables: author, book, borrower.

Explore denormalization, the process of intentionally introducing redundancy by merging previously normalized tables to boost query performance. Understand trade-offs of increased storage and anomalies during insert, update, or delete.

Explore dimensional modeling for analytical queries, detailing fact and dimension tables, star and snowflake schemas, and how they optimize query performance for a hotel reservation system.

Explore data storage and retrieval patterns, including data lakes, data warehouses, and data marts, containing raw data in original formats, with ETL/ELT, partitioning, indexing, and sharding for analytics.

Master partitioning, indexing, and sharding as they divide a table into partitions within the same database, speed up data retrieval with indexes, and distribute data across database instances by range.

Discover lambda and cap architectures, the five v's of Hadoop, and the key Hadoop ecosystem components. See how batch and stream processing yield accurate insights and real-time results.

Explore public, private, hybrid, and multi-cloud architectures, and learn how pay as you go cloud services from providers like AWS, Azure, and Google Cloud support secure, compliant data workloads.

Ingest raw JSON credit card transactions from S3, clean and validate data, mask PII, compute daily merchant summaries, and store in a data lake and MySQL for GDPR compliance.

Explore the credit card transaction life cycle from authorization to clearing and settlement, and learn how data engineering captures and prepares data for analysis and predictive modeling.

Design and implement a data architecture that ingests JSON data into S3, triggers a Lambda-driven EMR Spark ETL, masks PII, aggregates by merchant and date, and loads results to MySQL.

Set up AWS S3 data-lake for json and parquet, run Spark on EMR or PySpark, configure MySQL with RDS or local, automate with Lambda or Airflow, use GitHub with IntelliJ.

Design the Daily Transaction Store bucket on S3 with raw, cleaned, and aggregated data in year/month/day folders; use parquet for cleaned and aggregated, including merchant daily and customer spend.

Set up an AWS S3 bucket with versioning and a structured directory: raw/transaction/purchase, cleaned, and aggregated, then upload a raw file.

Upload raw data from an EC2 instance to S3 by provisioning an EC2, securing a PEM key, transferring files with scp, and using aws s3 cp to copy data.

Set up a git repository on GitHub, clone locally, and create a Python virtual environment. Open the project in IntelliJ, add and commit files, then push to main.

Activate the virtual environment in IntelliJ, install PySpark, configure the Python interpreter, run a spark job to create a spark session and data frames, and begin the ETL phase.

Define a data engineering project structure with extract, load, transform, and utilities folders, a Main.py entry point, and a db files directory for PySpark SQL statements.

learn to run the data pipeline with main.py by parsing arguments for mode (S3 or local) and date, then execute via spark submit to load data into S3 and MySQL.

Explore how main.py implements a data pipeline by configuring a spark session, creating Datareader, data cleaner, data masker, summary calculator, and writer objects, and executing the process_transactions method.

Master the singleton spark session manager that guarantees one spark session, uses the new method to get spark session, and stops it to clear garbage values.

Describe how the Config.py class loads credentials from credentials.env, enforces validation rules, masks card numbers, and exposes get jdbc url and get MySQL properties for MySQL access.

Datareader class initializes with a Sparksession and defines the predefined transaction schema. It reads from S3 and local paths, returning data frames and logging exceptions.

The data cleaner initializes validation rules, trims strings, uppercases currency, removes nonnumeric card characters, rounds amounts, formats timestamps, derives transaction date and r, validates data, returns valid and invalid frames.

Compute a completeness score with add data quality metrics by checking required fields, drop duplicates, then use clean and validate as the mother method to return valid and invalid dataframes.

Learn how the data masker class protects PII by masking card numbers to the last four digits, hashing customer IDs, and applying environment-based masking to produce a secure, minimal dataset.

The summary calculator class offers five methods: daily merchant summary, merchant risk matrix, hourly patterns, comprehensive summary, and merchant performance KPIs to deliver a 360-degree merchant view.

Utilize the data writer class to emit data frames to S3 (parquet or JSON), local JSON, and MySQL in overwrite mode, including invalid transactions, summaries, and data quality reports.

Learn how a data pipeline reads transactions from local or S3, cleans and validates data, applies masking, generates summaries, stores results in MySQL or S3, and ends with Spark shutdown.

Identify data quality issues in sample transactions, such as currency in lowercase, excessive decimal places, non-numeric card numbers, and duplicates, and review DDL for creating catalyst database in MySQL Workbench.

Submit the spark job to a local cluster, validate and mask data, generate daily merchant summaries and risk metrics, write transactions to MySQL, and generate a log report.

Inspect processed transactions (999 records) and a single invalid entry from a duplicate id 1006, then review daily merchant summaries and hourly patterns to debug and fix the data pipeline.