Campaign Response Testing

Learn how to "A/B test" your sales and marketing campaigns without hiring a data scientist or statistician.
4.1 (31 ratings)
Instead of using a simple lifetime average, Udemy calculates a
course's star rating by considering a number of different factors
such as the number of ratings, the age of ratings, and the
likelihood of fraudulent ratings.
3,867 students enrolled
Start Learning Now
  • Lectures 5
  • Length 1 hour
  • Skill Level Intermediate Level
  • Languages English
  • Includes Lifetime access
    30 day money back guarantee!
    Available on iOS and Android
    Certificate of Completion
Wishlisted Wishlist

How taking a course works


Find online courses made by experts from around the world.


Take your courses with you and learn anywhere, anytime.


Learn and practice real-world skills and achieve your goals.

About This Course

Published 4/2015 English

Course Description

Response based campaigns are sales or marketing campaigns where specific offers or advertisements are shown and we need to measure response rates per group. In this business problem we are choosing ad-copy and price points and are trying to pick the best combination. This is a traditional statistical problem and is often organized into what are called "A/B tests." However, more and more business professionals are required to directly plan and manage these sales and marketing campaigns without statistical or technical assistance.

This course will teach you how to plan and manage a response based marketing campaign. Learn:

  • What an A/B test is
  • How to pick campaign sizes
  • How expensive certainty can be

The method we are teaching is based on Bayesian statistical inference, so is a bit different than the more traditionally taught methods (t-tests, p-values). The advantage is the a number of natural business questions (such as how much money is at risk) are easier to encode in the Bayesian framework.

The course is unfortunately a very technical topic. To mitigate this we have wrapped the complete method in an interactive worksheet that we are sharing freely. The idea is: you supply the wisdom and judgement, the worksheet can perform all of the messy calculations.

The course structure is as follows:

  • Introduction to the course (free lecture)
  • Primer on the statistical terms needed for the course
  • Working with the worksheet
  • Review of the formal methods behind the worksheet
  • (optional) pointers to the code running the worksheet

The methods taught in this course should greatly improve your ability to plan and manage sales and marketing campaigns. You should be able to judge if this is the course for you from the free lecture.

The course largely describes a free Shiny app that performs campaign calculations for you. The app has a planning sheet and an evaluation sheet.

The purpose of the planning sheet is to use the user inputs (prior bounds on conversion rate and conversion value) to estimate an acceptable absolute error in campaign value. This acceptable error rate is used to campaign sizes that ensure the campaign chosen has a good probability of being close to the best choice in terms of relative error. A good way to use this is to enter two different rates you wish to be able to distinguish between on the planning sheet.

The evaluation sheet shows the posterior distribution of the (unknown and unobserved) true values of the campaigns conditioned on the observations the user has input.

What are the requirements?

  • Students should be familiar with basic statistics (probability, mean, variance, and the normal distribution), have some familiarity with sales or marketing campaign practices, and be willing to work with an interactive graphing worksheet.

What am I going to get from this course?

  • Learn how to choose the right size for sales and marketing campaigns and properly evaluate conversion rates of these campaigns.
  • Learn how to evaluate already run sales and marketing campaigns.

Who is the target audience?

  • This course is for business and technical professionals who have taken a statistics or data science course, but want to learn some more of the details the business application of experiment design and evaluation.

What you get with this course?

Not for you? No problem.
30 day money back guarantee.

Forever yours.
Lifetime access.

Learn on the go.
Desktop, iOS and Android.

Get rewarded.
Certificate of completion.


Section 1: Testing Campaign Response Rates

This lecture lays out the entire course. We define what problem we are going to solve (estimating the true response rate of your sales campaigns) and how we are going to solve it (with a free interactive worksheet). The free app is linked in the lesson extras/external-resources area. Also be sure to watch the introduction video (the video that plays if you click on the image when you first load this course).


In this lecture we will review some common technical terms from statistics. The goal is to make these terms less of a surprise when we use them.


This lecture is the meat of the course. We show how to work some problems in the campaign planning and testing worksheets. We freely share the worksheet and freely share some guide pages as downloadable materials.

4 questions

Try your hand at planning a sales campaign


Now that we have worked a few exercises together we review the statistical theory behind the worksheet.

14 pages

This optional slide deck gives pointers to the source code, references, and more.

Students Who Viewed This Course Also Viewed

  • Loading
  • Loading
  • Loading

Instructor Biography

John Mount, Data Scientist, Win-Vector LLC

I am principal at with the data science consulting firm Win-Vector LLC. Win-Vector LLC specializes in data science research, implementation, and training. I have over 10 years of experience in research, teaching, machine learning, and data science.

I am co-author of the popular book Practical Data Science with R, and I blog often on mathematics, programming, machine learning, and optimization on the Win-Vector blog.

My profesional experience includes managing a data science group for Shopping dot com (an eBay company), working in price optimization for Rapt (acquired by Microsoft), and apply machine learning at a web-scale for Kosmix (acquired by Walmart online). My original fields of study were mathematics (AB UC Berkeley) and computer science (Ph.D. Carnegie Mellon) with a heavy emphasis on probability theory.

Ready to start learning?
Start Learning Now