Ultimate AB Testing Course with Python Coding

Clarify the business problem before AB testing by identifying goals, stakeholders, KPIs, launch criteria, and required data, and align a team from sponsor to data scientist.

Define the goal of an A/B test within a cascading hierarchy from business mission to product goals to experimental levers, and design experiments that improve engagement and growth.

Coordinate a cross-functional team for AB testing by aligning the business lead, product manager, UI/UX designer, data scientist, and software engineer to plan, implement, and run experiments.

Define product kpi with the metric recipe—action, unit of analysis, and statistical function—and apply ARG and heart frameworks to align acquisition, activation, retention, revenue, and north star with guardrail metrics.

Learn how the north star metric (nzme) serves as the primary long-term kpi, aligning marketing, product, and engineering to deliver user value and profit growth.

Define the driver metric as a proxy for the NZME in AB tests, tied to a specific feature; use daily queries per user or daily minutes per user as examples.

Track guardrail metrics—business metrics and validity metrics—to evaluate tradeoffs and biases alongside the driver metric in ab testing, with examples like revenue per day, signups per day, and cancellations.

Define the three data aspects for experiments: data pipeline infrastructure for A/B tests, baseline performance with current KPI, and inference to decide whether to launch the change.

Participate in an exercise to define Northstar, guardrail, secondary, and segmentation metrics for an Amazon product search experiment with a new recommendation algorithm, and compare your solution with provided answer.

Learn to design an Amazon-style ab test for a product search recommendation system, selecting metrics like total sales and average daily sales per user, with guardrails and segmentation.

Analyze the pre-test data to define baseline performance, determine sample size and minimum detectable effect, and plan AB test duration using univariate analysis and visualizations.

Learn to state the experiment hypothesis before running an ab test. Define null and alternative hypotheses, set significance level and power, and minimum detectable effect at a 10% sign-up rate.

Define the business problem, craft the business hypothesis, and specify statistical hypotheses for AB testing. Compare control and variation with null and alternative hypotheses for KPIs like sign-up rate.

Define a business hypothesis in an AB test for a new recommendation algorithm, evaluating whether more relevant products shown to shoppers increase average order value per user.

Determine the best business hypothesis for an Amazon AB test by linking a KPI, the average order value per user, to the justification that showing relevant products will increase AOV.

Explore the statistical hypothesis for an Amazon product search experiment testing a new recommendation algorithm, and identify the best set of hypotheses from four options.

Present and justify the chosen statistical hypothesis: the null that the old and new recommender auvs are the same and the alternative that their obis differ, with explicit kpis.

Set the significance level (alpha) in A/B testing to reject the null hypothesis when the p-value is low, controlling for Type I error, with commonly 0.05, 0.01, or 0.10.

Explore how the significance level affects the type one error rate, using a 0.10 example and practice selecting the correct option before watching the solution.

Identify the type I error rate by linking it to the significance level, and conclude that a significance level of 0.10 corresponds to option C.

Explore how statistical power informs AB testing by comparing null and alternative hypotheses, alpha thresholds, and type II error, and learn how increasing sample size boosts power.

Evaluate your understanding of statistical power by relating it to a type II error rate of 0.10 and selecting the option among 0.10, 0.85, 0.90, or none of the above.

Assess understanding of statistical power by analyzing a test with a type II error rate of 0.10 and selecting the correct option. Watch the solution.

When an AB test shows a 0.06 p value with 0.05 significance and 0.80 power, do not reject the null; instead consider rerunning with higher power, e.g., 0.90.

Examine how minimal detectable effect defines the smallest statistically and practically significant difference between two groups, using absolute differences, relative differences or lift, and Cohen's d to quantify effect size.

Assess a reasonable minimum detectable effect for a large user base when testing a new recommender system to increase average order value.

Set plausible minimum detectable effect targets for an AB test of a recommender system, using AUV improvements on a mature platform with 100 million users, highlighting 1% gains.

Design the experiment for the ab testing course by selecting the randomization unit, determining how to randomize between control and treatment, and setting the per-group sample size and duration.

Randomization unit assigns users to control or treatment to remove bias and ensure comparable baselines before an AB test, typically 50/50, though not perfectly exact.

Learn how to compute sample size for A/B testing by balancing significance level, statistical power, and the minimum detectable effect, including variance, Z critical values, and one-tailed vs two-tailed tests.

Calculate the total sample size for a two-group ab test with significance level 0.05 and power 80%, given minimum detectable difference 10%, baseline mean 10, and variance 20.

Walks through a two-sample mean sample size calculation with 0.05 significance and 80% power, using delta 1 (10 to 11) and pooled variance 40 to reach 628 total.

Compute the experiment duration by multiplying the per-group sample size by the number of groups and dividing by daily allocated visitors. Account for 80% power, 7–14 days, and day-of-week effects.

Learn how to determine the percentage of total traffic to allocate for a two-group AB test over 14 days, with 10,000 users per group and 100,000 daily visitors.

Use the duration formula: duration equals total sample size divided by daily traffic times allocation percent; for 14 days, 20,000 samples, and 100,000 visitors per day, allocate about 1.43%.

Walk through code to design an AB test, calculating sample size and duration with power 0.80 and alpha 0.05, converting p one and p two to cone dx effect size.

Execute the experiment by exploring the experiment platform and infrastructure for running and scaling, and learn to peek at p-values and act when significance appears early.

Explore an experiment platform writes and runs AB tests, randomizes users, applies treatments, and tracks metrics with client and server side assignment, logs, and dashboards showing p-values and confidence intervals.

Avoid peeking at p-values in ab tests, because early stopping inflates the type I error rate beyond the 0.05 significance level when the null is true.

Run an ab test by randomly assigning users to control or treatment, compare signup rates, and evaluate significance over a 14-day study with 30,000 users.

Assess validity threats in experiments by examining internal and external threats, including SHRM and instrumentation effect, across the chapter’s video lessons.

Explain the stable unit treatment value assumption and how interference and spillover affect ab tests, with group-based or geography-level randomization to mitigate cross-group effects.

Explore survivorship bias in AB testing, where dropouts create incomplete data that bias control versus treatment comparisons, and note related approaches like survival analysis and causal inference.

Explain sample ratio mismatch in AB testing, including 50/50 splits during ramping, and detect faulty randomization with chi-square or one-sample z tests to ensure trustworthy results.

Explore the primacy effect in AB testing, where users resist a new change, causing a temporary KPI decline (change aversion), before adaptation leads to a rebound.

Novelty effect temporarily boosts KPIs like CTR when a new change is introduced, then fades and may inflate type I error; compare old versus new deltas or extend testing.

Identify the holiday effect as a temporary KPI boost during holidays that can inflate false positives; avoid running experiments in holiday periods for reliable results.

Learn how day-of-week variations, instrumentation integrity, and competitive activity influence AB test outcomes, and why running experiments for at least seven days stabilizes KPIs.

Learn how to assess AB test validity by performing an AA test, checking sample ratio mismatch between control and treatment with a chi-square test, and interpreting p-values at 0.05.

Learn how to conduct statistical inference, determine if a lift is statistically significant beyond 10%, and choose the right tests to evaluate results given the data.

Use the chi-square test of independence to assess whether personalization changes click-through rate, based on a 200-participant A/B test, observed and expected counts, and a 0.05 significance level.

Apply the chi-squared test of independence to compare control and treatment CTRs of 20% and 30% with 300 participants at 0.05 significance.

demonstrates solving a chi-squared test of independence for a control versus treatment ab test, building the contingency table and confirming a significant ctr difference at 0.05.

Assess the z-test for two proportions in a/b testing, comparing control and treatment groups using a pooled proportion, z score, p value, and confidence intervals.

Apply the z test for proportions to determine if the difference in CTR between control and treatment is significant at 0.05.

Use a z-test for proportions to compare control and treatment ctrs (150 each): 20% vs 30%. With p̂=0.25 and z=-2, reject null, concluding significant difference between generic and personalized emails.

Apply two-sample t tests for means to compare group differences, choose student or Welch based on variance, and interpret results with p values and 95% confidence intervals at 0.05 significance.

Apply a two-sample t-test to determine if the $5 increase in average order value from $25 (control) to $30 (treatment) is significant at 0.05, assuming equal variances of 4.0.

Apply a two-sample t-test for means to compare average order value per customer between old and new recommender systems, and conclude a significant difference at 0.05.

Conduct chi-square and proportion tests to evaluate AB test results and use confidence intervals to quantify uncertainty, concluding a significant lift from blue to green signup buttons.

Decide whether to launch an a/b test change by assessing lift, p-value, and confidence interval to determine statistical evidence for changing the button color from blue to green.

Understand how to interpret AB test results by converting absolute differences into relative differences (lift), illustrated with a signup rate example, p-value, and confidence interval.

Evaluate ab test results and decide whether to launch, rerun, ramp up, or not launch, considering kpis and secondary metrics and iterative tweaks.

Establish decision rules before experiments to map results to actions such as launching, rerunning, or scrapping ideas. Navigate scenarios using significance levels and practical impact with business intuition.

Leverage confidence intervals and p value to interpret AB test results, assess practical and statistical significance, and decide whether to launch, rerun, or stop experiments.

Analyze an A/B test on a sign-up page, changing the submit button from blue to green, then recommend launching the green button given a 12.8% lift and statistical significance.