Data Science Statistics A-Z : Python

Explore NumPy, the fast, array-based library for scientific computing and data analysis in Python, featuring homogeneous end-dimensional arrays, efficient memory use, and parallelized operations.

Import numpy as np in a Jupyter notebook to build arrays and matrices, inspect shape and dtype, and explore arange, zeros, identity matrices, diag, full, and random.

Explore NumPy operations in Python data science: create, index, reshape, and transpose arrays; perform boolean indexing, stacking, math functions, and linear algebra tasks like determinant, inverse, eigenvalues, and dot products.

Explore how the pandas package enables data analysis, data manipulation, and visualization, using data frames and cities to prepare data and support machine learning models.

Create pandas series from lists, explore homogeneous data types (integers, booleans, strings as objects), index and slice values, and generate date ranges with pandas date_range.

Learn to create pandas data frames from arrays, random data, and dictionaries using numpy; and load or save data from csv and Excel files.

Learn how to construct a data frame from a dictionary by naming variables such as temp, name, and random values, contrasting with a prior list-based approach.

Explore data frames in Python, inspect with head and tail, view shape and columns, access the name column by dot or brackets, and use describe and info.

Learn to set a data frame index from a column, sort by index or by column, and drop a column using in place or axis options to make changes permanent.

Explore how to use iloc for position-based indexing and loc for label-based selection in pandas, including row and column slicing, boolean masks, and combining conditions for data filtering.

Read csv data into a data frame in a Jupyter notebook, convert and analyze it, and save or read back from excel files using read_csv, to_excel, and read_excel.

Learn to merge data frames in pandas by concatenating d1 and d2 on a column called lesson number, handling missing values and controlling index or axis for rows or columns.

Explore groupby operations on a data frame, grouping by event, viewing group details, applying functions and lambdas to temperature data, and generating new columns like hot_temp.

Create two sample data frames with city and weather-related columns, then merge them on the city key using pandas' merge, exploring left and outer joins.

Explore pivot tables in pandas as an Excel-like way to create aggregate tables. Compute the mean temperature by event to show how temperatures vary across events.

Explore how linear algebra underpins machine learning by using vectors, distances, dot and cross products, and projections to analyze n-dimensional data in Python.

Explore matrices and vectors, identify rows and columns, and learn square, diagonal, and identity matrices, then practice matrix addition, multiplication, and transpose in a Python notebook.

Explore linear independence and dependence among matrix columns, determine rank, and use determinants and inverses to assess and remove redundant features in data analysis for machine learning.

Learn how to represent lines in 2d and planes in 3d as linear equations, and extend to hyperplanes in n dimensions using weight vectors and the X vector.

From two-dimensional to n-dimensional, explore how a normal vector defines a plane, compute point-to-plane distance, and extend circle equations to hyper-spheres.

Learn how inferential statistics use probability theory to draw population conclusions from sample data, illustrated by exit polls and quality checks.

Explore probability theory with coin and dice examples, defining sample space and events, and use random variables like X to quantify outcomes in a bank default scenario.

Construct a probability distribution for the random variable X, the number of red balls drawn, from 75 trials and plot a histogram to illustrate the outcomes 0–4.

Explore how to compute the expected value from a probability distribution using a red balls example, deriving the mean and its implications for game outcomes.

Compute the expected value by weighting outcomes (150 rupees gain, -10 rupees loss) with probabilities 0.133 and 0.867, yielding 11.2, indicating a loss on average; discuss adjusting prizes or penalties.

Discover how to use the binomial distribution to calculate probabilities without experiments, with step-by-step examples of drawing red and blue balls and applying product and combination rules.

Explore the binomial distribution by modeling n trials with binary outcomes, using n choose k, p, and 1−p to compute probabilities for x successes.

Compute cumulative probability by summing the distribution’s values up to a chosen x, such as P(X ≤ 1), and compare with the corresponding graph.

The lecture explains using bins to estimate probabilities for continuous data, compute cumulative probability (CDF), and compare probability density functions with histograms and areas under the curve.

Explore the normal distribution with mean mu and standard deviation sigma, and apply the 68-95-99.7 rule to estimate probabilities around mu. Calculate ranges like mu±2sigma and mu±3sigma.

Standardize data with the z score by computing (x - mu)/sigma, as in (43.25 - 35)/5 = 1.65. Read the z-table for the cumulative probability of that value.

Explore sampling distribution through sample means, see larger samples narrow the curve, and learn the standard error as sigma divided by the square root of n.

Learn the central limit theorem: with sample size above 30 the sampling distribution is normal; the mean approaches the population mean, and the standard deviation is divided by sqrt(n).

Learn how to construct a confidence interval for the population mean using the sample mean, sample standard deviation, and n, via the margin of error and the central limit theorem.

Compute a 99% confidence interval for lead in Maggie from a mean of 2.3 ppm and a standard deviation of 0.3, yielding about 2.22–2.38 ppm, below 2.5 ppm.

Explore hypothesis testing as a tool to infer population parameters from samples, differentiate it from general inferential statistics, and validate claims with real-world examples like food safety thresholds.

Formulate null and alternate hypotheses to guide hypothesis testing. Decide when to reject or fail to reject the null, illustrated by innocent versus guilty cases.

Explore hypothesis testing by formulating null and alternative hypotheses using equality, inequality symbols, and terms like at least and at most, illustrated with sales and air conditioners as examples.

Formulate null and alternative hypotheses and decide to reject or fail to reject using one-tailed and two-tailed tests, with critical values and directional considerations.

Apply the critical value method to test hypotheses about the monthly average units sold, using known population standard deviation and the sample mean to decide on rejecting the null.

Evaluate evidence against the null using a z-table, compute the standard error from population standard deviation and sample size, and compare to 95% confidence bounds to decide on rejection.

Explore hypothesis testing with a one-tailed test, defining null and alternative hypotheses, computing the critical value and z-score, and deciding whether to reject the null at 5 percent significance.

Learn to perform a one-sided hypothesis test for a lead content limit, choosing alpha, computing the critical value, and deciding whether to reject the null or take no action.

Explore how the p value measures evidence against the null hypothesis, relates to the 0.05 significance level, and how to compute and interpret two-tailed tests using the corresponding score table.

Learn how type I and type II errors arise in hypothesis testing, with alpha and beta tradeoffs, through practical examples and business considerations.

Explore the t distribution, its symmetry with the normal distribution, and how unknown population standard deviation leads to using degrees of freedom (n-1) to compute p-values and consult tables.

Use the t distribution for small-sample, two-tailed tests with unknown standard deviation; apply df = n−1 (24 for n=25, 31 for n=32) and compare to 0.05 critical values.

Demonstrate transforming raw data into graphs with Matplotlib and Seaborn. Visualize data using line plots, scatter plots, box plots, and histograms, with labels, axes, and subplots.

Explore seaborn to visualize distributions, box plots, pair plots, and heat maps, uncovering correlations, outliers, and multivariate patterns in Python data science workflows.

Use box plots to compare sales and profit by product category and customer segment. Analyze mean and median and log transformations to reveal profitability patterns and shipping costs influence.

Convert the order date to datetime, compute daily sales, and visualize time series with seaborn, including pivot tables and heat maps of monthly means by year.

Learn to perform exploratory data analysis on user data, uncover patterns, and generate actionable business insights as you learn data science basics.

Explore steps to prepare data for analysis, focusing on data sourcing and cleaning, and address issues like missing values encoded as xx or 9 9 9 and city name changes.

Explore univariate analysis on a single column to identify patterns, then examine two or more columns together, and create derived metrics to count outcomes across groups.

Compare public and private data types and sources. Learn how to fetch and prep public data for geographic and economic analysis, and navigate related challenges.

Explore banking and telecommunication data use cases, focusing on credit card data and delinquency indicators, where data is sensitive; discuss churn analysis, plan optimization, and promotional decisions.

Explore data sourcing and cleaning across sectors, from employer turnover signals and missing information to retail stock, pricing, and cross-selling insights, and media advertising.

Explore public data sets from government portals and sector-specific sources, such as agriculture and finance, and learn how to locate, access, and reuse data for research and development.

learn to clean public election data by converting to text format, fixing alignment and spelling, addressing missing and invalid values, and standardizing and renaming columns for analysis.

Identify and fix missing data, standardize units and formats, treat invalid values, and remove duplicates to prepare clean, comparable data for analysis.

Master univariate analysis of data types by exploring unordered and ordered categorical data, numerical data, and metadata, using frequency plots, rank frequencies, and log-scale visuals to reveal patterns.

Explore univariate analysis of numerical data, reviewing mean, median, mode, and standard deviation, and learn to handle outliers using median, quartiles, and box plots for unit rate analysis.

Perform segmented analysis by grouping data and applying summary statistics such as mean and median, then compare groups using box plots and hypothesis testing.

Explore how bivariate analysis uses two variables to reveal relationships through correlation, including positive and negative trends and both numeric and categorical data insights, with real examples.

Learn the art of deriving new columns to improve machine learning models by exploring type-driven, business-driven, and data-driven metrics, with practical examples and data transformations.

Learn how machine learning algorithms learn from data to solve real-world problems across industries. See examples from voice assistants, search autocomplete, health care bed planning, and banking churn prediction.

Explore regression, classification, and clustering as core machine learning types, and understand supervised and unsupervised learning with outputs that are continuous or categorical.

Explore linear regression as a key predictive model in supervised learning, comparing simple and multiple regression while predicting future outcomes across industries such as finance, medicine, and sports.

Understand how simple linear regression links marketing spend to sales, defining independent (predictors) and dependent (target) variables, and estimating the line y = beta0 + beta1 x.

Explore how to derive the best fit line by selecting beta0 and beta1 to minimize the residual sum of squares, using errors, predictions, and squared errors.

the lecture compares residual sum of squares and total sum of squares, and defines r squared, the coefficient of determination, as 1 minus rss over sst.

Investigate simple linear regression linking TV marketing budget to sales, import data into Python, visualize the relationship with a scatter plot, and validate with train/test splits.

Explore a simple linear regression case study by comparing actual and predicted values, inspecting residuals, and evaluating performance with mean squared error and R-squared.

Explore residual square error (RSE) and the residual sum of squares (RSS), its relation to degrees of freedom, and why RSS comparisons matter for evaluating models.

Explore a data science project on investment analysis, focusing on data cleaning, exploratory plots, sector and country analysis in english-speaking markets, with no machine learning involved.

Import and inspect the data, clean and encode the datasets, normalize company names to lowercase, address missing values, and fix encoding issues to produce a clean, analysis-ready dataset.

Load and verify data, build a clean data frame. Describe missing values, apply inner joins to merge master data, and impute or remove data to retain about 77 percent.

Perform funding, country, and sector analyses for investments between 5 to 15 million, comparing venture, seed, angel, and private equity using median emphasis and outlier visualization in English-speaking countries.

Clean and merge sector data from the mapping table, normalize category names to lowercase, and create a unified sector column for sector analysis and investment visualization.