Data Cleaning and Preprocessing using Python

Compare Jupyter Notebook, Jupyter Lab, and Google Colab, noting cloud-based access, GPU support, Drive integration, and pre-installed libraries, with trade-offs like internet dependence and offline use.

Install and configure Anaconda to manage Python packages with conda, set environment variables, and launch JupyterLab via Anaconda Navigator for data cleaning and preprocessing in Python.

Learn to use jupyter lab, the open source web interface for notebooks that mix text, images, and code, with run, debug, and extensions.

Create a table of contents from a notebook by clicking the left sidebar menu. Use hashtags to mark titles and subsections, and adjust numbering and depth settings to unfold sections.

Learn to format text in notebooks with markdown, including bold, italics, underline via HTML, strikethrough, colors, equations, line breaks, and lists.

Explore how the Anaconda console supports creating and managing virtual environments, installing packages, and controlling Python versions with commands like set location, get-childitem, and conda list.

Compare conda and pip as Python package managers, learn to identify which library belongs to which, view versions, and install specific versions with conda's single equals and pip's double equals.

Update libraries one by one to avoid incompatibilities and security risks. Check compatibility with Conda info and pip show, and keep a record of libraries, including versions and builds.

Create and manage Python environments with conda and pip to prevent version conflicts, install libraries, and test projects. Activate, clone, deactivate, list, and remove environments using conda and Anaconda navigator.

Explore Python data types and dynamic typing, covering integers, floats, scientific notation, complex numbers, booleans, strings, date objects, and essential conversions, formatting, f-strings, isoformat, and strftime.

Explore Python arithmetic operators—addition, subtraction, multiplication, division, floor division, modulus, exponentiation, negation. See how precedence and parentheses alter results for detecting periodic data and generating sequences.

Explore Python's comparison operators, including equal, not equal, greater than, less than, greater or equal, and less or equal, and how they drive if and while loops and string comparisons.

Master Python's logical operators and, or, not to combine conditional statements, control program flow, and evaluate complex tests within if, while, and for blocks.

Explore assignment operators in Python that assign values to variables and perform mathematical and logical operations, simple assignment, addition, subtraction, multiplication, division, exponentiation, and bitwise and with the bin function.

Explore identity operators in Python, including is and is not, to check if variables reference the same object in memory, not just equal values, with examples using lists and None.

Learn how Python membership operators in and not in test presence of values in lists, tuples, dictionaries, sets, or strings, and use them for data filtering, key verification, and validation.

Create and manipulate Python lists to store ordered, mutable data of different types, use indexing, slicing, membership testing, and concatenation for flexible data preprocessing.

Explore common list methods in Python, including append and extend, remove, pop, insert, index, count, reverse, copy, clear, and sort with length-based and ascending or descending options.

Learn to traverse list elements with for loops using range, len, and enumerate, print each item and access specific indices, with while loop control and handling multiple lists.

Learn to replace for loops with list comprehensions to square numbers, filter even elements with if and else, and build new lists from existing data.

Explore lists of lists in Python to store tabular data, access elements with brackets, and use list comprehension to extract columns or filter fruits by price.

Explore pandas series as a time series, a one-dimensional, indexable data structure, create them from lists or dictionaries, and use methods like value_counts and basic statistics to handle missing values.

Explore Python tuples, created with parentheses or without, learn their immutability, how to determine length and elements, and perform unpacking, indexing, and use of count and index methods.

Learn to modify a tuple by converting it to a list, updating an element, converting back, then use the sorted function to order ascending or descending via reverse, and reassign.

Learn how to pack tuple elements into variables and access them by position, then concatenate two tuples using the plus operator in Python to form a new combined tuple.

Explore Python sets: unique, unordered collections with curly brackets; learn len, membership testing, and core methods such as add, update, remove, discard, union, intersection, and difference.

Learn how Python sets help deduplicate lists and reveal intersections in data science, from countries of origin to registered vs purchased users, using f-strings to display results.

Explore dictionaries, a flexible key-value data structure that stores diverse data types and remains modifiable. Create them via curly brackets or the dict constructor, with unique keys and repeatable values.

Explore Python dictionary operations, including clear, pop, update, and delete, and learn how to add, modify, and retrieve keys and values efficiently.

Access information in Python dictionaries using keys and nested dictionaries; index lists and use get to avoid errors. Use loops or list comprehension to flatten languages and collect professions.

Explore NumPy arrays as a core data structure in scientific programming, data science, and machine learning, enabling vectorized operations, slicing, masking, and concatenation.

Create and query numpy structured arrays with named fields to store heterogeneous data, convert lists of lists to tuples, assign dtypes, and filter rows by conditions.

Learn how to save numpy arrays on your computer in binary using pickle or in text format with JSON, including serialization, reading, and reloading arrays.

Explore dataframes, pandas' two-dimensional data structure, with rows, columns, indices, and headers. Learn to convert lists to dataframes, access columns and rows using loc and iloc, and view values.

Learn to append and merge dataframes using pandas pd.concat in Python, performing vertical concatenation with ignore_index to avoid duplicate indices, and horizontal concatenation with axis=1 to align columns.

Learn to access and transform data in pandas DataFrame with for loops and list comprehensions, then convert a dictionary to DataFrame and set the index from the names column.

Convert a dictionary into a dataframe using pandas, enabling tabular data handling, missing data management, vectorized operations, and seamless integration with visualization and machine learning tools.

Learn how the Python if statement uses conditions, elif and else branches, with indentation to execute code like printing whether someone is an adult based on age.

Explain the Python while loop, including condition, increment, modulus tests for even/odd numbers, and using break to avoid infinite loops.

Explore how to use the for loop to iterate over ranges and lists, print sequence values, and access indices with enumerate and f-strings.

Learn how to define and use user defined functions in Python using def, parameters, and return to modularize code, invoke functions with arguments, and perform tasks like greeting and addition.

Explore the difference between local and global variables in Python, focusing on scope and memory lifetime. Learn how the global keyword enables modification inside a function and variable persistence.

Explore user defined functions in Python with advanced parameters, including default values, optional args, variable length args, and keyword dicts, using examples with the className parameter, students, and user profiles.

This lecture explains higher-order functions, showing how they take functions as parameters and return functions, reducing code repetition through a printing example and a greeting generator in Python.

Learn how Python's map function processes and transforms every element of an iterable—such as a list, tuple, or dictionary—without explicit loops, by applying a function, converting results to a list.

Learn how Python's filter function selects iterable elements with a test function, using even numbers as an example, and see how combining filter with map can improve efficiency.

Explore Python lambda functions as small anonymous expressions, using them with map and filter to perform operations like addition, joining words, squaring numbers, and conditional transformations.

Learn how Python generator functions use yield to yield values and save state, enabling memory-efficient, one-at-a-time processing of a sequence of data with for loops.

Explore exception handling in Python using try, except, else, and finally to catch value and division errors, keep programs running, and perform cleanup.

Learn to debug Python code with PDB breakpoints, inspect variables, and step through loops to fix a division-based logic error, restart the kernel, and verify correct multiplications.

Learn to debug Python code in Jupyter Lab with the Source Python kernel and a dedicated conda environment, enabling advanced debugging panels that show variable values and code flow.

Master visual debugging in JupyterLab by stepping through a multiplication function, setting breakpoints, exploring the call stack, and fixing an integer division bug to prevent zero results.

Classify variables into qualitative and quantitative, with nominal and ordinal qualitatively, discrete and continuous quantitatively, and interval and ratio scales, illustrated by age, weight, temperature, and money.

Import a csv file using pandas readcsv, handle separators and quotes, detect encoding with chartered, and read the file with the correct encoding for preprocessing.

Learn how to import csv files with pandas read_csv, choose delimiters, set index columns, and selectively load columns while cleaning european income formats for numeric analysis.

Discover pandas data types such as int64, float, object, bool, category, and datetime64, learn to handle nulls, set dtypes, and use to_datetime for accurate data analysis.

Learn to read and create excel files with pandas using read_excel and to_excel, handle single or multiple sheets, check shape, use usecols and iloc, and manage sheets with sheet_name.

Learn to create and read parquet files with pandas and pyarrow, explore columnar storage benefits, include metadata, and selectively load columns and filtered rows for big data workflows.

Learn to create, read, and convert JSON files into dataframes for analysis, leveraging Python's json library to store web API data and machine learning hyperparameters.

Convert nested json into a dataframe using json normalize, flattening the transactions list into rows and producing a seven column table with client and purchase data.

Convert csv data to json and tailor its structure for APIs using pandas tojson with orient options, including records and values for matrix operations or applying machine learning techniques.

Connect Python to a Microsoft SQL Server with SQLAlchemy and pandas, create an engine, authenticate, query the Ciudad table, load results into a dataframe, and view the first ten rows.

Learn to inspect and correct data types in pandas, converting yes/no to boolean, parsing dates with a defined format, and converting risk to a categorical for reliable preprocessing.

Explore category-based typo error correction in Python data cleaning, detecting erroneous categories, converting appropriate columns to categorical, inspecting unique values, and using a replacement dictionary to fix errors.

Detect and correct typographical errors in text with a python spell checker, including interactive prompts and case-insensitive corrections.

Correct typographical errors in a dataframe's text across multiple columns using a spell checker and a phrase-correction function. Apply corrections with pandas apply and display the final dataframe.

Normalize text in tabular data to unify operating system categories, lowering case, trimming spaces, removing accents, and mapping synonyms with a dictionary in pandas.

Detect and remove duplicate records in a video games data set using pandas, count duplicates with the duplicated method, and reduce from 1097 to 1093 rows with drop_duplicates.

Learn to clean data by removing noise, fixing typographical errors and incorrect formats, and addressing duplicates and inconsistencies to prevent garbage in, garbage out.

Identify out of range values in lists and dataframes using a range dictionary. Replace them with nulls and locate the out of range values by column and row indices.

Explore data cleaning and preprocessing with Python by identifying logical inconsistencies in dates and quantities. Build and compare dataframes, detect invalid dispatch, and display results in Jupyter using IPython display.

Identify and fix invalid dates by converting strings to pandas date format with to_datetime, handling errors with errors='coerce', and flagging out of range dates using min date and max date.

Identify non-numeric entries in numeric-looking columns, use the info method to inspect dtypes, convert with pandas to numeric using coercion, then filter nulls to detect remaining non-numeric rows.

Detect malformed emails using regular expressions and a dataframe of addresses; apply str_match with na = false to identify invalids, invert results with tilde, and filter by .com or .net.

Identify outliers, atypical values distant from the data, and learn to detect and manage them. Explore their impact on mean, variance, regression, and model performance.

Study how the normal distribution underpins univariate outlier detection, highlighting its bell shape, symmetry, equal mean, median, and mode, and the 68-95-99.7 rule and applications.

Assess normality by visualizing with a histogram and a normal quantile-quantile plot, and apply Kolmogorov-Smirnov or Shapiro-Wilk tests; if p > 0.05, conclude the variable is normally distributed.

Generate a 100-point dataset with mean zero and unit standard deviation using numpy, then visualize with matplotlib hist and qq plots and test normality with Kolmogorov-Smirnov and Shapiro-Wilk from SciPy.

Detect outliers using the standard deviation by comparing values to the mean with a two- or three-standard-deviation threshold. Validate the approach with a histogram and the Kolmogorov-smirnov test for normality.

Learn how to handle outliers in Python data cleaning by detecting values within two standard deviations, replacing or imputing with NaN and bounds, and preserving originals with copies.

Standardize variables to a mean of zero and unit variance using z scores, enabling cross-scale comparisons and model compatibility for linear regression, k-means clustering, and neural networks.

Detect outliers by standardizing data with a z-score, verify normality using a Kolmogorov–Smirnov test, and apply a z threshold of ±3 (or ±2 for stricter) using sklearn's StandardScaler.

Detect and handle outliers using z-score bounds of -2 and 2, replace with nulls or reverted lower/upper values, and restore data by undoing standardization with mean and std dev.

Detect outliers with the interquartile range and visualize results using a horizontal box plot in Python. Compute Q1, Q3, and IQR, apply 1.5 times IQR whiskers, and interpret outliers.

Detect and replace outliers using the interquartile range in Python, converting extreme values to np.nan, then replacing them with the lower and upper whisker limits and verifying the results.

Explore DBSCAN, a non-parametric, density-based clustering method that identifies outliers and forms clusters of any size and shape without predefining the number of clusters, using point density and neighborhood proximity.

Learn how DBSCAN uses epsilon, min samples, and distance metrics to identify clusters and outliers, with KNN aiding epsilon selection and a density-based core point concept.

Learn to detect and handle outliers with DBSCAN in Python by computing distances with k nearest neighbors, choosing epsilon via elbow plot, and visualizing results.

Detect multivariate outliers with DBSCAN on a 3D dataset, using k-nearest neighbors, elbow-based epsilon, and 3D plotting to distinguish blue cluster points from red outliers.