A/B Testing 101

Develop intuition for statistics behind A/B testing by exploring probability, variability, sample sizes, and confidence without formulas.

Explore how variability introduces uncertainty in A/B testing, and learn how standard deviation helps compare control and test outcomes to measure real effects.

Explore how variability and sample size shape confidence in results using standard deviation and running averages. See how more data stabilizes estimates of the true average for a/b testing.

Apply stats intuition to A/B testing by balancing variability, standard deviation, and sample size, with confidence levels, t-tests, and chi-squared tests to judge differences.

Build statistical intuition for A/B testing by examining variability, standard deviation, and sample size, then apply confidence levels and tests like t-test and chi-squared to determine real differences.

Plan effective A/B tests by choosing what to test, selecting target metrics, estimating duration, and building a solid test plan, then interpret results to guide decisions.

Define the top-level goal, explore performance metrics and underperforming segments to identify opportunities, then research, brainstorm, and prioritize test ideas before building an A/B testing plan.

Estimate how long to run an a/b test by calculating required observations. Account for day-of-week seasonality, trend data, and practical constraints to plan experiment logistics and avoid bias.

Select a target metric by balancing your top line goal and opportunity, choosing among category menu click through rate, conversion rate, and revenue per user for testing practicality.

Formalize your experiment by detailing the hypothesis, target audience, confidence level, target metric, lift threshold, run duration, and budget, while reducing variability and ensuring proper tracking.

Learn when to check A/B test results, balance educated peeking with QA and ramp-up to protect against false positives and test budget, and interpret significance at your chosen confidence level.

Learn how to interpret A/B test results using hypothesis testing, 95% confidence, and daily trends, then make bias-aware decisions with a structured review process.

Identify high-opportunity tests, plan to reduce variability, determine sample size and timing, select a target metric, and interpret results with bias control to make informed decisions.

Explore advanced concepts in A/B testing, including A/B/C experiments and single-study variant comparisons. Learn to handle skewed metrics, normal-data issues, and use segmentation with conversion-rate metrics.

Identify and avoid common A/B testing errors, including misreading business conditions, p hacking, and metric selection. Prioritize short-term metrics over long-term targets to ensure reliable results and minimize bias.

Avoid A/B testing when you will implement regardless, when results take too long, or when you cannot run two versions simultaneously, or when the test is overly complex.

Explore alternatives to A/B testing, including before-and-after testing, market testing, data analysis, and research, with their advantages, limitations, and how to use them alongside A/B experiments.

Wrap up course on A/B testing by engaging in the Q and A for instant answers, posting questions, and rating the course, and complete lectures and quizzes for the certificate.