Math 0-1: Probability for Data Science & Machine Learning

Explore probability basics, including definitions, simple rules, and the distinction between probability and statistics. Use Venn diagrams and probability trees to illustrate probability, Bayes rule, independence, and conditional probability.

Define probability as the frequency of occurrence, a proportion, illustrated by a fair coin's heads or tails and long-run proportions approaching 50%, reflecting our uncertainty about outcomes.

This lecture shows proportions, like six heads in ten flips, are estimates—not the definition of probability. It distinguishes probability as reasoning about uncertainty from statistics, which applies probability to data.

Using an urn with blue, green, and yellow balls shows why the equal-probability definition is flawed; the correct definition quantifies uncertainty from the system, not via data.

Build probability models by defining a sample space omega, events as its subsets, and a probability measure p that assigns values and enforces additivity for disjoint events, p(empty)=0 and p(omega)=1.

Visualize probabilities with venn diagrams, mapping the sample space and events as sets. Apply complement, intersection, union, and De Morgan's laws in set notation and boolean logic.

Explore properties of probability models using Venn diagrams, derive the complement rule P(A^c)=1−P(A), and prove union and subset relations: P(A∪B)=P(A)+P(B)−P(A∩B), with P(A)≤P(B) when A⊆B, and the corollary P(A∪B)≤P(A)+P(B).

Apply the union rule p(a∪b)=p(a)+p(b)-p(a∩b) to a fair die, with a=even and b=greater than three, yielding p(union)=2/3.

Apply the law of total probability by partitioning the sample space into disjoint events and summing p(ai ∩ B) to get p(B), illustrated with hearts in a deck.

Explore conditional probability and how partial information updates uncertainty. Learn the p(a|b) rule, p(a∩b)=p(a|b)p(b), and the conditional form of the law of total probability, with Covid and die examples.

Learn Bayes' rule for conditional probability, including P(B|A) = P(A|B)P(B)/P(A), and its role in Bayesian statistics and machine learning, with marginal probabilities and the law of total probability.

Apply Bayes’ rule to a rare disease test, using 0.1% prevalence and 99% accuracy, to compute the probability of disease after a positive result.

Explore independence in probability by contrasting dependent variables like hair and gender with independent die and coin outcomes, and apply probability via p(a∩b)=p(a)p(b). Differentiate mutual independence from pairwise independence.

Explore mutual independence with a four-event example, deriving pairwise, three-way, and four-way conditions where p(a∩b)=p(a)p(b), p(a∩b∩c)=p(a)p(b)p(c), and p(a∩b∩c∩d)=p(a)p(b)p(c)p(d).

Explore probability tree diagrams to list all outcomes of a sequence and compute joint, conditional, and total probabilities, with coin flip and market examples illustrating multiplication and sum rules.

Define probability as a measure of uncertainty and explore sample space, events, and probability measures. Apply rules for union, intersection, complement, conditional probability, Bayes rule, and independence.

Apply a practical suggestion box to gather learner feedback for math 0-1 probability for data science and machine learning, including background, course, difficulty, missing explanations, and future topics.

Introduce discrete random variables and probability distributions, define random variables, and survey Bernoulli, binomial, geometric, and Poisson distributions with real-world binary outcomes like coin tosses and ad clicks.

Define random variables as functions mapping each outcome in the sample space to a real number, enabling convenient probability distributions.

Explore the Bernoulli distribution from the coin flip, where a discrete random variable is 0 or 1 with theta as the probability of success, and learn its pmf and support.

Explore the categorical distribution with k categories, where probabilities theta1 through thetaK sum to one, and use them to define p(x) and indicator-function representations.

Explore the binomial distribution by counting successes in n coin flips, deriving P(X=k)=n choose k theta^k(1-theta)^{n-k}, and using the binomial theorem to show the probabilities sum to one.

Explore the geometric distribution through coin flips, modeling the number of tails before the first head with parameter θ, and verify the distribution sums to one.

Explore the Poisson distribution as the limit of the binomial with large n and small p. Derive p(x) = lambda^k / k! e^{-lambda}, with lambda as the Poisson parameter.

Visualize discrete probability distributions in Python by plotting pmfs for Bernoulli, Binomial, Geometric, and Poisson with bar charts, using pre-written code to focus on concepts.

Explore discrete random variables and their distributions, including Bernoulli, binomial, geometric, Poisson, and categorical distributions, with PMFs, supports, and the role of theta as probability of success.

Explore continuous random variables and probability density functions, contrasting them with discrete mass functions, and learn to integrate common continuous distributions—uniform, normal, exponential, Laplace—to show area one.

Explore the distinction between discrete and continuous random variables, learn why probabilities are assigned to ranges using the probability density function, and contrast PMF with PDF.

Explore a physics-inspired view of probability density, treating probability as mass distributed in space. Learn how a density function rho and spatial integration reveal probability in a region.

Explore continuous probability distributions by linking pdfs, pmfs, and cdfs, and learn to normalize a quadratic pdf by integrating from 0 to 1 to find c=3.

Explore the uniform distribution, its flat density on [0,1], and how sampling and the normalization constant c = 1/(b-a) shape the density on any interval [a,b].

Explore the exponential distribution, derive the pdf lambda e^{-lambda x} for x >= 0, and discuss the normalizing constant. Apply to modeling arrival times for buses, customers, and other events.

Explore the normal (gaussian) distribution, the bell curve, and how mu and sigma shape its center and spread, used for modeling grades and errors, with support minus to plus infinity.

Explore the Laplace distribution as a practical prior in machine learning and bayesian statistics, derive its pdf, define its center and scale, and verify the distribution integrates to one.

Visualize continuous distributions in Python with code to see how pdfs change with parameters, focusing on xvals, linspace, and location and scale across uniform, exponential, normal, Laplace, gamma, and beta.

Explore continuous random variables and their probability distributions, focusing on PDFs, uncountable ranges, and key distributions like uniform, exponential, normal, and Laplace, with the mean and variance.

Learn CDFs from PDFs and PMFs, change of variables, and distributions of functions of random variables, including joint, marginal, and multivariate normal cases for sampling.

Explore cumulative distribution functions (CDFs) as a tool for discrete and continuous distributions, showing the CDF at x equals P(X ≤ x) and that it ranges from 0 to 1.

Compute the cdf of the geometric distribution by summing p(k) = (1-θ)^k θ up to n, deriving F(n) = 1 - (1-θ)^{n+1}, with floor for real inputs.

Learn how to derive CDFs from PDFs for continuous distributions, including uniform, exponential, and standard normal, using the fundamental theorem of calculus and practical numerical methods or tables.

Learn to compute the normal cdf for any mu and sigma by transforming to the standard normal, using phi to obtain the cdf.

Master change of variables for random variables, from discrete to continuous, by mapping y = h(x) and summing p(x) over h^{-1}(y) with die roll and Bernoulli examples.

Derive the pdf of y from a continuous x in a function of a random variable using change of variables, inverse functions, and absolute values, addressing increasing and decreasing cases.

Explore how joint distributions model the relationship between two random variables, using the joint CDF and marginal distribution to derive probabilities for x and y.

Define and compute joint pmfs and pdfs for discrete and continuous variables, derive marginals, and explore the two-coin toss example and bivariate normal.

Demonstrate that the marginal of a bivariate normal is normal with the same mean and variance, and explain how marginalization ignores correlation rho.

Condition on random variables across discrete and continuous cases, derive conditional and joint distributions, marginalization, and Bayes' rule for probability and density functions.

Derive the conditional distribution of y given x for a bivariate normal, showing y|x is normal with mean mu2 + rho sigma2/sigma1 (x − mu1) and variance sigma2^2(1 − rho^2); reveals linear regression properties.

Explore independence of random variables, distinguishing independence from zero correlation, and apply joint and marginal factorizations for discrete and continuous cases, including iid.

Explore the bivariate normal with zero correlation and show that rho equals zero makes x and y independent by factoring the joint pdf into the product of their marginals.

Explore multivariate distributions and random vectors, focusing on continuous and discrete cases, including the multivariate normal and multinomial, with notes on independence, conditioning, Bayes rule, and the Wishart distribution.

Explore the multivariate normal distribution, also called mvn or vector gaussian, focusing on the vector x, the mean mu, the covariance sigma, and its compact pdf with a quadratic form.

The multinomial distribution generalizes the binomial to multiple categories, modeling counts from n trials with category probabilities theta across D categories, and applying to document modeling with word counts.

Demonstrates how the two-dimensional multivariate normal reduces to the bivariate normal pdf by deriving the covariance matrix, its inverse, and determinant, and clarifying the roles of rho, sigma1, and sigma2.

Demonstrates that in a multivariate normal distribution, zero covariance between variables implies independence, and the joint pdf factorizes into independent marginals.

Explore multidimensional change of variables in the discrete setting, transforming a joint pmf pXY to pZW via z=h1(x,y) and w=h2(x,y), covering 1-to-1 and non-1-to-1 cases with a Bernoulli example.

Explore multidimensional change of variables in the continuous case, derive the joint pdf using the Jacobian, and invert the transformation to show how uniform samples map to standard normals.

Explore discrete and continuous convolution, clarifying the difference from cross correlation in deep learning, and show how the sum of independent random variables leads to convolution of their distributions.

Derive that the sum z = x + y of jointly normal random variables is normal, with mean mu_x + mu_y and variance sigma_x^2 + sigma_y^2 + 2 rho sigma_x sigma_y.

Plot cdfs for uniform, exponential, normal, laplace, and other distributions in Python. Visualize a mvn multivariate normal to study correlation rho and covariance effects on joint distributions.

Derive CDF from PMF and PDF, and recover full distributions for discrete and continuous variables. Explore marginal and conditional distributions, independence, Bayes rule, multivariate distributions, change of variables, and convolution.

Explore the expected value as a linear operator, derive mean, variance, and moments, and connect them to correlation, conditional expectation, reinforcement learning, and the Bellman equation.

Explore the expected value, or mean, for discrete and continuous distributions, and learn the e of x notation, mu as the mean, and the center-of-mass intuition.

Explore the expectation operator and the average value of g(x) for discrete and continuous cases, including joint distributions, linearity, independence, and covariance as an application.

Explore variance as a measure of spread, derive Var(X)=E[X^2]-E[X]^2, and learn how scaling by a affects variance and standard deviation.

Practice computing the mean and variance of the Bernoulli distribution from its pmf and expected value, noting the mean theta and variance theta(1−theta).

Compute the mean and variance of the Poisson distribution from first principles. Show that the mean equals lambda and the variance also equals lambda using its pmf p(k)= lambda^k e^{-lambda}/k!.

Derive the mean and variance of the normal distribution by transforming the PDF with t=(x−μ)/σ, applying integration by parts, and showing the results μ and σ².

Compute the mean and variance of the exponential distribution with rate lambda by applying integration by parts to the pdf, deriving E[X]=1/lambda and Var(X)=1/lambda^2.

Explore skewness, kurtosis, and moments as measures of distributions, including their third and fourth standardized moments, and how heavy tails imply more extreme values than the normal.

Examine the kurtosis of the normal distribution by standardizing x to z, computing the fourth moment, and using integration by parts to show kurtosis equals three.

Explore covariance and correlation, define their formulas, and prove key identities. Learn how independence implies zero covariance and build the covariance matrix for random vectors.

Show that the covariance of x and y in a bivariate normal equals rho sigma1 sigma2, so the correlation is rho.

Apply a concrete exercise that shows zero correlation does not imply independence by using x uniform on -1 to 1, y = x^2, and computing covariance.

This lecture demonstrates that correlation measures linear dependence, yielding a perfect correlation for a linear y = a x + b; it also shows y = x^2 has zero correlation.

Define and compute conditional expectation for discrete and continuous cases, using a joint Bernoulli distribution to show how e[x|y] depends on y through p(x|y) and p(y).

This lecture extends conditional expectation to the continuous case, proving linearity for conditional expectations and deriving E[X|Y=y] from a joint distribution, showing it is a function of y.

Learn the law of total expectation, proving that E[E[X|Y]] = E[X], with continuous and discrete forms, and explore reinforcement learning links to the Bellman equation.

Explore the distribution of a linear combination of jointly normal variables, derive its mean and variance, and preview moment generating functions for extending to more variables.

Compute the mean and variance of a weighted sum y = w^T x for any distribution, with mean y = w^T mu and variance y = w^T sigma w.

Explore the expected value, mean, and the linearity of expectation, then study variance, covariance, correlation, moments, skewness, kurtosis, and conditional expectation with the law of total expectation for reinforcement learning.

Explores generating functions, including the moment generating function and the characteristic function, and shows how they help compute sums of independent variables and advance toward the central limit theorem.

Define the moment generating function as E[e^{tX}]. Differentiate to obtain moments; not all variables have an MGF (CF always exists), but an MGF uniquely defines the distribution.

Compute the moment generating function of the exponential distribution with rate lambda, derive that MGF is lambda/(lambda-t) for t<lambda, and obtain mean 1/lambda and variance 1/lambda^2.

Derive the normal moment generating function, from the standard normal to mu and sigma squared, yielding the function exp(t mu + 1/2 sigma^2 t^2) and the mean mu, variance sigma^2.

Define the characteristic function phi_X(t) = E[e^{i t X}] and show its link to the moment generating function and Fourier transform, with derivatives at zero generating moments.

Illustrates an example where the moment generating function does not exist for a distribution with density 1/x^2 on x ≥ 1, showing the MGF diverges and distribution has no moments.

Derive the normal characteristic function by substituting z = (x - mu)/sigma, completing the square, and obtaining phi_x(t) = exp(i mu t - 1/2 sigma^2 t^2).

Explore sums of independent random variables by using the moment generating function, showing that the sum's mgf equals the product of individual mgfs, with Bernoulli and binomial examples.

Compute the mgf of Poisson and describe the sum of two independent Poisson variables, which is Poisson with parameter lambda1 plus lambda2.

Explore how to compute the distribution of the sum of independent random variables using mgfs and see how summing geometric variables yields the negative binomial distribution.

Derive the moment generating function for a random vector and apply completing the square to the multivariate normal. Show M_X(T)=exp(mu^T T + 1/2 T^T Sigma T) and verify by differentiation.

Explore characteristic functions for random vectors, akin to mgf, but with i in front, defined as phi of x of t as expected value of exp(i t^T x) for vectors.

Compute the mgf of a weighted sum of jointly normal variables and show that w^T x is normal with mean w^T mu and variance w^T Sigma w.

Demonstrate generating functions in code to derive the sum distribution of independent exp(1) variables using characteristic functions, inverse Fourier transform, and convolution.

Explore moment generating functions and characteristic functions for random variables and vectors, including sums of independent or iid variables and the uniqueness theorem.

Learn inequalities involving probabilistic quantities and expectations, deriving Markov and Chebyshev inequalities. See their use in risk management, finance, quality control, and machine learning with UCB.

Explore monotonicity of expected value: if x ≤ y, then E[x] ≤ E[y]. Proof sets z = y − x and uses linearity and non-negativity for discrete and continuous cases.

Bound the probability of non-negative random variables exceeding a threshold using Markov inequality, via two proofs, and set the stage for Chebyshev's inequality.

Present Chebyshev inequality: P(|Y − μ| ≥ a) ≤ Var(Y)/a^2, and interpret how variance controls deviation from the mean, with a Markov-based path to the law of large numbers.

Apply the Cauchy Schwarz inequality to vectors and random variables, showing |cov(x,y)| ≤ sqrt(var(x)var(y)) and that correlation lies between -1 and 1 via a quadratic discriminant.

Introduce key probabilistic inequalities, including monotonicity, non-negativity, and the Markov inequality, and use them to prove Chebyshev and Cauchy–Schwarz for correlation bounds.

Explore the law of large numbers, showing the sample mean converges to the expected value with data, and the central limit theorem, which makes sums of random variables approach normality.

We define convergence in probability for a sequence of random variables, using the notation X_n ->^p Y, and show that P(|X_n - Y| ≥ ε) → 0 as n → ∞.

The weak law of large numbers states that for independent random variables with the same mean mu and finite variance, the sample mean converges in probability to mu as n grows.

Examine convergence with probability one, or almost surely convergence, where X_n converges almost surely to y and implies convergence in probability, with the strong law of large numbers as application.

the strong law of large numbers states that for iid variables with mean mu (variance not required), the sample mean converges almost surely to mu, implying convergence in probability.

Explore the frequentist view by linking the indicator function's expected value to the probability that x lies between a and b, via the law of large numbers.

Explore convergence in distribution for variables, defining x_n converges to x in distribution via their cumulative distribution functions at continuity points, and compare with convergence in probability and almost surely.

Explore the central limit theorem: sums of iid random variables with finite mean and variance converge to a standard normal when properly standardized, regardless of the original distribution.

Explore how the law of large numbers and the central limit theorem unfold in Python by generating synthetic data, computing sample means and variances, and examining distribution shapes.

Learn limit theorems and three convergence types—convergence in probability, almost sure convergence, and convergence in distribution—and how the weak/strong laws and central limit theorem connect them.

Explore the gamma distribution, its alpha and beta parameters, and the gamma function, including the gamma pdf and its non-negative support, with bayesian machine learning applications.

Study the beta distribution, a 0-1 model for rates like click-through and conversion, with alpha and beta parameters and its relation to the beta and gamma functions in Bayesian analysis.

Learn how the chain rule of probability factorizes a joint distribution into a product of conditional distributions. It applies to sequences and underpins transformers, rnn language models, and time series.

Explain why the normal distribution integrates to one by using the joint pdf of two independent standard normals, applying change of variables to polar coordinates, deriving Rayleigh and uniform components.