Complete Course on A/B Testing with Interview Guide

Explore product experimentation concepts, learn about different types of experiments, and understand the end-to-end testing process with templates to prioritize tests and select the best experiment for a given situation.

Explore the three types of experiments: A/B testing, multivariate testing, and multi-armed bandit testing, and learn how each compares, when to use them, and how to assess statistical significance.

Explore multi-armed bandit testing as a dynamic, data-driven alternative to A/B and multivariate tests, reducing regret by reallocating traffic to top variants to maximize conversions.

Explore when to use A/B testing versus multi-armed bandit testing, highlighting data collection with statistical confidence and real-time conversion optimization.

Learn the four-stage process of product experimentation and A/B testing, from define and design to implement and analyze, including monitoring metrics, goals, hypotheses, variations, and results.

Explore why statistics underpin a/b testing, including statistical hypothesis testing and significance, and learn how analyzing a sample predicts outcomes for the remaining audience to optimize conversion.

Explore how sampling error and variance affect A/B testing results, define population and sample, and learn regression towards the mean to improve estimating the true conversion rate.

Explore how confidence intervals bound the population mean around the sample mean, and how margin of error, sample size, and confidence level shape their width.

Use confidence intervals in AB testing to assess whether variation B improves variation A. Test the null hypothesis and determine statistical significance by examining the difference between population means.

Explore p values and statistical significance in a/b testing, interpreting results under the null hypothesis, with alpha settings to reduce type I errors.

Explore statistical power, its link to type II errors and beta, and how significance level, sample size, and effect size shape power in A/B testing.

Explore how to determine sample size and duration for AB tests, applying product experimentation insights, experiment types, and core statistical concepts to guide test decisions.

Account for statistical significance and power when planning A/B test sample size and duration, and consider business cycles, novelty effects, and seasonality with a predefined end point.

Join section four to consolidate all lessons through a practical example, detailing the full process from conceptualization to analysis in R.

Design and analyze an A/B test for a lamp promotion in an app, from hypothesis generation to sample size, power, and metrics like transaction rate, purchase value, and uninstall rate.

Visualize data with histogram and density plots using ggplot2 and dplyr, check randomization through baseline comparisons, then assess treatment vs control on purchases and uninstalls to decide not to implement.