Optimization & A/B Testing Statistics

Speed up your a/b tests by doing it right from the start.
2.9 (38 ratings)
Instead of using a simple lifetime average, Udemy calculates a
course's star rating by considering a number of different factors
such as the number of ratings, the age of ratings, and the
likelihood of fraudulent ratings.
1,129 students enrolled
Take This Course
  • Lectures 21
  • Length 3 hours
  • Skill Level All Levels
  • Languages English
  • Includes Lifetime access
    30 day money back guarantee!
    Available on iOS and Android
    Certificate of Completion
Wishlisted Wishlist

How taking a course works


Find online courses made by experts from around the world.


Take your courses with you and learn anywhere, anytime.


Learn and practice real-world skills and achieve your goals.

About This Course

Published 5/2012 English

Course Description

Whether you've got a lean startup or a fat Fortune 500, the faster you learn the faster you'll grow.  Optimization and a/b testing is at the heart of learning fast.

I guarantee you will learn something in this course that will raise your skill level.  With the 30-day money-back guarantee, you can't lose. 

We start with the basics, then cover the 8 steps of running a solid a/b test.  Next we dive deep into the statistics behind hypothesis testing.  In the long-run you will save your organization headaches by setting up tests correctly and analyzing them with the right statistical rigour. 

There is double and triple digit ROI around optimization for companies that figure it out.  Start now and impress your colleagues on Monday morning.

Topics include:

Examples of a/b tests

Hypothesis testing

Measurement as risk reduction

Selecting a KPI or success metric

8 Steps for Running an A/B Test

Selecting from amongs a/b test and MVT test designs

Lift Threshold

Null Hypothesis

Statistical significance

Sample size estimates

confidence interval

test statistic


standard error of the mean


Fischer Exact test

Statistical Power

Type I error

Type II error


How to choose what statistical test to run

What you get with this course?

Not for you? No problem.
30 day money back guarantee.

Forever yours.
Lifetime access.

Learn on the go.
Desktop, iOS and Android.

Get rewarded.
Certificate of completion.


Section 1: Introduction

Madmen of today should be called mA/Bmen since they ply experimentation, not hard liquor from 9 to 5.

Learn why and how to apply a/b testing to your marketing efforts.  We'll start at the beginning, with the foundations of the science behind hypothesis testing.


A quick intro what organizations gain from optimization programs and where it can be applied.


We cover two different a/b tests to provide a starting point for what a test really looks like.


We cover some of the special language of optimization.  A good reference to come back to.


You probably know there is a difference.  But we apply this to optimization and explore the options.

Section 2: Goals & Test Designs

Get this wrong and the hard work that went into the experiment is lost.   Choosing appropriate success metrics in an A/B test is a bit more nuanced than many think.


How do you setup your test when you care about optimizing to more than one outcome?


We cover the basic test designs you might consider when planning out your experiment and the pros and cons of each.


What is Measurement vs Hypothesis Testing?

Section 3: Testing in 8 Steps
8 Steps to A/B Testing

Writing a precise statement is Step 1.


Determing Sample Size needed.


Run until sample size reached.

Section 4: Statistical Tests
One Sample T-Test
Two Sample T-Test
Standard Error & Confidence Intervals

Learn to select the right statistical test for your experiment. (Part 1)

Select Stat Test (part 2)
Chi Square (part 1)
Chi Square (part 2)
Section 5: Wrap-Up

Students Who Viewed This Course Also Viewed

  • Loading
  • Loading
  • Loading

Instructor Biography

Jared Waxman, The Marketer's Mythbuster: A/B Testing & beyond

Jared Waxman works at Yahoo! in Sunnyvale CA where he serves as Senior Director of Growth.

Previously he worked for Adobe in San Jose CA, where he served as Group Manager for Online Optimization & Analytics for Adobe.

In his nearly twenty years of experience in Silicon Valley where he gained experience in website optimization, web analytics, and growth hacking he has been on both the tools-side and the client-side, beginning as Director of Product Management for Alexa Internet's web measurement division, before working with Amazon and Intuit. He holds a Bachelors of Science from Yale University.

Ready to start learning?
Take This Course