From Beginner to Data Engineer: Your Ultimate Learning Guide

Join Neeraj as he introduces Python for data science, explaining beginner-friendly syntax and open source benefits. Explore essential packages like scikit-learn and pandas through hands-on exercises.

Learn the theory of Python operators, including arithmetic, comparison, and logical operators, with examples of operands and how true/false values determine results.

Explore arithmetic, comparison, and logical operators in Python within a data science context. See how operands affect results, handle type mismatches, and observe booleans in a Jupyter notebook.

Discover how variables bind to objects in Python using the assignment operator, and how int, float, bool, and str types are stored. Learn naming rules, case sensitivity, and type flexibility.

Explore Python variable assignment, naming rules, and data types for data science, using type checks and demonstrations of case sensitivity and object references.

Explore how to implement conditional decisions in Python using if, elif, and else, with proper indentation and comparison operators, illustrated by practical data science examples.

Implement conditional statements in Python through hands-on Jupyter Notebook exercises, mastering if-else and if-elif-else to print even or odd numbers and assign grades.

Master Python looping constructs for data science using for loops and iterables like range to repeat tasks efficiently. Explore how Python's for loop differs from other languages.

Explore implementing for loops in Python by using range to print numbers from 11 to 49 and to print odd numbers between 10 and 50; learn range behavior and stepping.

Learn how Python functions encapsulate reusable code, including built-in and user-defined functions, defined with def, arguments, and return, and how to call them to compute a circle's area.

Learn to implement functions in python by building a compare function that returns the greater of two numbers, using def, an if statement, and proper indentation.

Examine the flaws of Python's primitive data types for mixed data and compare how lists store mixed values in sequence while dictionaries store mixed data types, though they are unordered.

Learn how lists store ordered, indexable elements to handle sequences efficiently, using slicing, negative indices, and operations like append, extend, remove, and del, with for loops to iterate.

Learn to implement lists in Python by creating lists, indexing and slicing, appending and extending, removing by value or index, and iterating to perform simple elementwise operations.

Learn how dictionaries store data as key value pairs, an unordered data structure, and access elements by keys, not indexes, using adding, updating, and deleting operations in Python.

Learn to implement dictionaries in Python using a Jupyter Notebook: create and access key-value pairs, add entries with assignment or update, delete with del, and note case sensitivity.

Explore how Python's standard library, modules, and packages organize built-in functions, data types, and file handling, and learn import techniques to reuse code efficiently.

Explore why pandas is essential for data handling in python, learn how to read csv and excel files, and work with data frames for basic analysis.

Import pandas and read a CSV file data.csv with read_csv, loading into a dataframe named df; then read an Excel file data.xlsx with read_excel, and verify with df.head().

Learn how dataframes in pandas compare to excel sheets and perform basic operations such as checking shape, viewing head and tail, and accessing columns by name or list.

Learn to load data with pandas using read_csv and inspect a dataframe's shape, head, tail, and columns, then select single or multiple columns such as bare nuclei and cancer type.

Explore how to index, select, and subset data in pandas, including column operations, row slicing with iloc, condition-based filtering, and exporting to csv and Excel file formats in a notebook.

Sort dataframes with pandas using sort_values and sort_index to order by one or more columns, in ascending or descending order. Explore in-place options and restoring the original order with sort_index.

Learn how to merge Pandas data frames with concat and merge, performing outer, inner, left, and right joins, and understand row-wise and column-wise concatenation.

Use the apply function in pandas to perform row-wise and column-wise data preprocessing on the Big Mart sales data set, faster than a for loop, enabling feature engineering.

Explore aggregating data with pandas, using group by, crosstab, and pivot tables on the Big Mart sales dataset, to compute mean prices by item type and mean sales by outlet.

Master the basics of matplotlib in a Jupyter notebook for data visualization by creating and customizing plots, adding labels, legends, colors, figure sizes, ticks, and subplots.

Visualize the Big Mart sales dataset with Matplotlib, using various plots. Compute mean price by item type and mean sales by outlet size to reveal insights.

Explore seaborn basics, a matplotlib-based high-level library that creates graphics from pandas data frames, enabling one-line plots like line, bar, histogram, box, violin, and scatter, with color by a variable.

Master Seaborn data visualization with advanced categorical plots using catplot, covering strip, swarm, box, violin, bar, and KDE density plots.

Learn how regular expressions identify text patterns for feature engineering, including matching names starting with su and three letters after, and extracting digits with \d.

Explore how Python regular expressions work in practice, using raw strings and the re module to match, search, and find all patterns, with examples on emails, dates, and titles.

Explore the fundamentals of databases and why we need them, including relational and NoSQL types, data storage, and real-world MySQL usage.

Explore why databases matter in the data age, where back end storage, instantaneous retrieval, and fast search support the four operations: store, update, delete, and search.

Explore what a database is as an organized collection of data stored and accessed electronically from a computer system, and how formal design and modeling techniques support efficient querying.

Explore the core properties of a good database: scalability to handle growing traffic, elasticity for on-demand capacity, security and governance to protect data, and data integrity for accuracy.

Compare relational databases with NoSQL databases, noting tables with fixed columns versus schema-free JSON documents stored in collections, and review examples like MySQL, Postgres, MongoDB, and Redis.

Explore how relational databases store data in tables, using a user details table with user id, name, and skills, and link rows through relationships. Next, explore NoSQL databases.

Explore how NoSQL databases store data using JSON, with key-value pairs, nested documents, and lists, and compare this flexible approach to relational databases.

Explore why 80% of companies use MySQL, and see NASA, Boeing, Bank of America, Apple, and Google among leaders across aerospace, finance, government, technology, and internet sectors.

Learn to install and configure MySQL, access remote MySQL servers, and understand client-server architecture, MySQL distributions, and local installations on Mac, Windows, and Linux, plus licensing and GUI clients.

Explore client–server architecture through real-world examples like the internet, where a browser sends requests to a server and receives responses via a communication network, including MySQL client–server interactions.

Compare MySQL distributions 5.7 and 8, highlighting roles, multilingual support, and performance improvements, and note MariaDB’s compatibility as a fork with local Mac, Linux, and Windows installs requiring admin rights.

Install MySQL 5.7 on Mac by downloading the installer, starting the server from System Preferences, and accessing it via terminal with mysql -u root -p, or use mysql dump.

Install and verify a MySQL server on Ubuntu using apt and sudo, then access the MySQL shell and note the 5.7 server version and MySQL dump.

Install and configure MySQL on Windows using the official installer, set a strong root password, and access the MySQL shell to manage the server.

Explore licensing of MySQL Community Edition, including its GPL three open source model, free use for non-commercial development, and options to commercialize through source changes or purchasing a commercial license.

Connect to a remote MySQL server with host, username, password, and required permissions, using the plain SQL command with -h, and ensure the MySQL client is installed.

Explore graphical user interfaces for MySQL, compare free and paid GUI clients, and use the MySQL shell to reinforce module two installation and configuration concepts.

Discover SQL for data science basics, MySQL queries, DDL/DML/DCL, create tables, insert and select data, and understand MySQL data types and NULL versus NOT NULL.

Discover what SQL is: Structured Query Language, a standard database language used to create, maintain, and retrieve relational databases, and to communicate with the database across hardware.

Trace history of SQL from 1972 when Edgar Court at IBM proposed a language for databases, through Chamberlin and Boyce's work on IBM System R, to 1979 SQL trademark change.

Connect to MySQL from a terminal using mysql -u root -p and enter your password. The video covers the MySQL monitor, version 5.7.24, GPL license, and command types to come.

Explore ddl, the data definition language that defines the database schema. Learn how create, drop, and alter commands define and modify database objects.

Explore DML commands for manipulating data in databases, including select, insert, update, and delete, using SQL statements in the course from beginner to data engineer: your ultimate learning guide.

Explore data control language (DCL) and how it manages rights and permissions in database systems, including grant and revoke commands that control user access, with optional TCL coverage to follow.

Create a MySQL database for a job portal, run your first SQL query, and verify with show databases before using the job portal database for future table creation.

Learn to create tables in MySQL by defining a table schema named user details with fields name, designation, and salary. Verify creation with show tables and prepare to insert records.

Learn how to insert data into a MySQL table using the insert into command, specifying columns name, designation, and salary, and verify results with select star from user details.

Learn to use the select statement to retrieve all data or specific columns from a user details table, and apply where clauses to filter by salary or designation in MySQL.

Explore MySQL data types, including integer with length, varchar, text, timestamp, and enum for categorical choices, plus float and blob.

Explore how null vs not null constraints enforce mandatory fields, how to set not null during table creation or with alter, and how inserts fail when required fields are missing.

Update data using the update command, delete data using the delete command, describe the current structure of a table, and alter the table to modify database structures.

Learn to update data in a table with the update command, updating one column at a time, using a where clause to target rows, or update all rows if omitted.

Master the update command in MySQL shell to modify designation and salary in the user details table, performing updates one column at a time in two steps with where clauses.

Master the delete command by using delete from table name where condition, noting that omitting the where clause deletes all rows, and plan to practice in the MySQL shell.

The delete command removes records; three rows with salaries greater than or equal to 50 are deleted, and never run delete without a where clause to avoid deleting all rows.

Explore two MySQL commands: describe shows a table's schema, while show create table returns the create statement to recreate the table; a practical demo follows.

Describe and show create commands in the MySQL shell to inspect the user details table, view fields and nullability, and generate the create table statement with engine and charset notes.

Learn how to alter tables in MySQL: add a column, rename it, modify data types, and drop columns, using describe to verify changes.

Master importing data from CSV files into MySQL, exporting tables to CSV, and backing up databases in SQL format for easy restoration on another server.

Learn to import a csv file into MySQL using load data local infile, mapping csv columns to the user details table, ignoring the header and using a comma delimiter.

Export data from a MySQL table to csv using select * from user_details and into outfile '/tmp/user_details.csv', with fields terminated by and optionally enclosed by, then verify the file content.

Learn how to back up data from MySQL with MySQL dump, creating sql files for the job portal database and single-table backups, with insert statements and integrity checks.

Learn how to restore databases from backups, including creating a blank database, using SQL files, and handling errors when a database is deleted or moved between servers.

Learn data analysis fundamentals for data science, including counting rows, aggregation, and finding key statistics. Slice, filter, sort, group, and roll up data to derive insights and identify extreme values.

Count rows with the count operator on the user details table, returning six. Use the distinct keyword to find five distinct salary values.

Learn how to use MySQL aggregation functions—sum, average, and standard deviation—to compute basic statistics, rename results with aliases, and analyze salary distributions.

Learn to identify extreme values by using SQL, applying select max and select min on the salary in user details.

Learn to slice data with sql where clauses using in, not in, and between, and combine and or to sum salaries for sales and CEO groups.

Shows how to limit data with limit and offset on a user details table, retrieving the top two records or skipping the first two rows.

Learn how to sort data to find the top earning user by salary using order by, descending, and limit, and optionally select the name.

Master filtering patterns with the like operator and wild cards to query data by starting with, ending with, or containing characters; use percent and underscore in MySQL with examples.

Learn to use group by in MySQL to roll up data, compute department distributions, and aggregate salary metrics, with having, order by, and where filters.

Explore descriptive analytics on the FIFA 19 players dataset, learning data eyeballing, data dictionary, and how to build, load, and analyze in MySQL with simple and advanced queries.

Perform data eyeballing on the FIFA 19 csv dataset in Excel to explore 16,000 rows and 15 columns, noting attributes like id, age, nationality, ratings, club, value, wage, joined date.

Understand what a data dictionary is and how it describes each variable in a dataset, using the FIFA 19 example with fields like id, name, age, nationality, value, and penalties.

Frame the right business questions and extract actionable insights from the FIFA 19 dataset by analyzing player counts, nationalities, wages, ratings, club performance, and date-based distributions using MySQL.

Analyze data and design a MySQL table structure by assigning fields to data types, then create FIFA 19 database and players table while noting indexing trade-offs and preparing import CSV.

load data from csv into the MySQL players table using load data local infile, handle csv delimiters and header row, and review warnings and date format considerations.

Analyze the FIFA 19 dataset to compute player counts, nationalities, wage totals and averages, top nationalities, and top clubs by total and average ratings using simple SQL queries and subqueries.

Explore advanced data analysis with queries on players' foot distribution, the luckiest jersey by total wage, nationalities by club prefix, and date-based joins with year-on-year trends.

Learn to retrieve data from multiple tables using joins and subqueries, including self joins, with hands-on mysql shell examples.

Explore why joins are essential in MySQL and relational databases by retrieving data from multiple tables, illustrated with a job portal example using user, job, and applies tables.

Explore the main join types in MySQL, including left join, inner join, right join, and Cartesian (cross) join, with insights on usage frequency and practical guidance.

Learn how the left join combines all records from the left table with matching records from the right table, using a common key, with syntax and practical examples.

Learn the left join with a practical retail example: join purchases (left) with contact details on user id to fetch names and phone numbers of active customers.

Learn how the inner join returns only matching records from both tables, how you can chain inner joins, and use a syntax similar to the left join.

Learn how the cross join creates a Cartesian product by pairing every row of the first table with every row of the second, a computationally heavy operation that's rarely used.

Explore the right join, rarely used and often replaced by the left join; it returns all right-table records with matched left-table rows, using a simple syntax.

Explore self joins in sql by joining a table to itself using aliases, to generate pairwise lists such as players from the same club or age differences under ten.

Explore indexing to speed queries in MySQL using syntax and concepts such as primary keys, foreign keys, and auto increment constraints on tables, and examine database relationships and their types.

Learn how indexing speeds up search and queries in MySQL by using single or composite indexes, weighing trade-offs with insert and update times.

Learn how indexing speeds queries by organizing data like a library, using b-trees to sort records, and how insertions and updates impact performance.

Explore how relationships in sql connect the user details, job details, and applied tables in a job portal database through id_user and id_job.

Learn about one-to-one, one-to-many, and many-to-many relationships with examples like a user and a social security number, multiple profiles, and job applications, modeled with primary keys and foreign keys.

Explore table constraints in MySQL, including primary keys for unique row identification, foreign keys for relational links, uniqueness, autoincrement, and one-to-one as well as one-to-many relationships.

Learn to use the concat function to join strings, such as building a full name from first and last names, and concatenate name with nationality using a pipe separator.

Learn how to convert string cases, including uppercase and lowercase, with UCS and LCS examples on nationalities from the players table.

Learn how to trim strings by removing leading and trailing spaces with the trim function, right trim, and rtrim, and apply trim to columns or variables.

Learn how to slice strings by extracting substrings with left, right, and substring functions, including taking the first two characters and last two characters from names in a players table.

Learn how to obtain the current date and time using date time functions, separate the current date and time, and generate the Unix timestamp with simple queries.

Learn to extract date and time components from a field with the date function, including month, day, year, and hour, minute, second, and explore related MySQL date functions.

Learn to format date and time as strings using a date_format function, customize display with formats like mm/dd/yy, and consult MySQL documentation for more options.

Learn numeric functions for rounding and formatting numbers, including round, floor, ceiling, and truncate operations. Apply abs to remove negative signs and control decimal places, such as two decimals.

Master manipulating MySQL data from Python by setting up a virtual environment, installing the MySQL connector, and performing select, insert, search, delete, and indexing operations.

Set up a virtual environment to isolate Python installations for projects, and learn to create, activate, and deactivate it with virtualenv on Mac, Windows, Linux, to connect Python to MySQL.

Install the MySQL connector to connect Python to MySQL with pip, requiring internet access. Ensure MySQL server or client is installed and running; consult module resources for config-file issues.

Learn how to connect MySQL from Python using the MySQL connector, test the environment, and configure host, username, password, and the FIFA 19 data set database to establish a connection.

Connect to MySQL with a cursor, run a select query on the players table, and fetch the top ten rows as a list of tuples for Python-based analysis.

Learn how to create a MySQL table with name and phone number from Python, insert a row using parameterized SQL, and commit the transaction with basic error handling.

Learn to delete records in a MySQL database using Python by writing a delete from contacts query and calling db.commit. Understand basic error handling in production databases.

Learn to query a MySQL database with Python using the where clause to filter players by nationality in the United States, returning names as a list of tuples.

Learn how to create and drop an index on a MySQL table using Python, including using cursor.execute and committing changes to optimize queries on the contacts.name field.

Explore the basics of cloud computing, including data centers, virtualization, on-demand compute, storage, networking, and global access via pay-as-you-go models.

Explore public cloud, private cloud, and hybrid cloud deployment models and learn when to use each based on shared resources, pay-as-you-go costs, and security.

Explain SaaS, PaaS, and IaaS as cloud service models, detailing what each provider or user manages, with examples like Google Workspace, Elastic Beanstalk, and AWS.

Learn how to create databases and collections in MongoDB, list databases with show dbs, and switch using the use command. Discover implicit collection creation when inserting a document.

Learn to insert documents in MongoDB, singly and in bulk, using insert one, insert many, and insert methods in the demo database user collection, with the _id field.

Learn to read documents from a MongoDB collection using find, filter by employee ID and name, and view results with pretty or find one.

Explore how the MongoDB _id field acts as a unique primary key for documents, how ObjectId values are generated, and how ordered versus unordered inserts work.

Import and export data in MongoDB using mongo import and mongo export commands, including JSON arrays and outputs in JSON or CSV.

Back up MongoDB databases and collections with the Mongo dump command, and restore to an existing or new database using the Mongo restore command, specifying the database, collection, and path.

Learn how to update documents in MongoDB with update one and update many, using set, increment, and rename operators to modify fields and add new ones.

Master how to manage MongoDB data by deleting documents with delete one and delete many, removing fields with unset, and dropping collections and databases.

Learn to perform CRUD operations on the MongoDB Atlas cloud cluster, including loading sample data, querying collections, and inserting, updating, and deleting documents.

Import and export data with MongoDB Atlas using mongo import and export. Create or append to databases and collections, handling json arrays, csv, and tsv with header and fields options.

Explore big data through the five v's—volume, variety, velocity, veracity, and value—and how diverse data sources like emails, logs, and social media drive real-time insights.

Explore the major challenges of big data, including storage, processing, velocity, and variety, and derive insights by ensuring veracity and integrating multi-source data with suitable tools.

Explore how big data powers fraud detection in banks, real-time recommendations in OTT platforms, and predictive analytics across retail, law enforcement, healthcare, and home security.

Explore how distributed systems overcome traditional scalability and downtime challenges for big data, using replication, horizontal scaling, fault tolerance, and the Apache Hadoop framework.

Discover how Hive, a data warehouse on top of Hadoop, offers SQL-like querying with HiveQL, translating queries into MapReduce jobs for scalable batch analysis of data, not a relational database.

Explore Hive as a data warehouse on Hadoop, featuring schema on read and a simple Hive Query Language that converts queries to MapReduce jobs for analysis.

Explore Hive's core components, including clients and interfaces, driver, compiler, metastore, and execution engine, and how they collaborate with Hadoop's HDFS and YARN to run MapReduce query plans.

Connect to Hive via beeline from the IT labs, using the provided connect command and your username, then set your Hive metastore warehouse directory to your own Hive warehouse.

Create databases in hive using the create database command, optionally with if not exists and a comment, then use show databases and describe database to view location in the warehouse.

Explore Hive data types by examining primitive types (numeric, date/time, string, and boolean or binary) and introduce complex types like arrays, maps, and structures with practical examples.

Learn how to write a hive create table command for reading formatted data with map, struct, and array types. Define fields, collections, and map key separators.

Learn how to create a Hive table with complex types (map, struct, array), define delimiters and storage as text, and understand Hive warehouse directories in HDFS.

Learn to load data into Hive tables by uploading, unzipping, and placing files in HDFS, then run load data in path into table and describe for schema details.

Compare managed and external tables in hive, noting metadata in the metastore, data in the hive warehouse or external locations, and drop behavior differences.

Learn to create an external hive table by using the external keyword and location to point to an external directory in HDFS, with data such as employees.

Learn how to create external tables from existing external tables and create managed tables from existing tables in Hive, using create external table, like, location, and create table as select.

Drop tables in hive using the drop table command for both managed and external tables. Verify with show tables and hive warehouse checks, noting that external data remains.

Master Hive alter table commands to rename tables, add or rename columns, and drop unused columns, with examples using show tables and describe.

Learn how to start generative AI on AWS, understand costs and free tier limits, set up SageMaker, and use Bedrock models for speech, text, and image tasks.

Set up your AWS environment by selecting optimal regions, exploring SageMaker and Bedrock, and obtaining access to Titan and open-source models for hands-on data engineering and AI workloads.

Learn to set up Amazon SageMaker domains in the correct region, create a VPC, and launch notebooks with Jupyter Lab or unified studio, including git clone workflows.

Learn to set up and run Jupyter notebooks in SageMaker, clone code from GitHub, and develop with foundation models, embeddings, and end-to-end data engineering workflows.

Explore multimodal embeddings by matching image and text descriptions using cosine similarity. Learn to generate, modify, and compare images and descriptions with Bedrock, SageMaker, and AWS cognitive services.