Campaign Response Testing
3.9 (47 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
4,250 students enrolled
Wishlisted Wishlist

Please confirm that you want to add Campaign Response Testing to your Wishlist.

Add to Wishlist

Campaign Response Testing

Learn how to "A/B test" your sales and marketing campaigns without hiring a data scientist or statistician.
3.9 (47 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
4,250 students enrolled
Created by John Mount
Last updated 5/2015
Price: Free
  • 44 mins on-demand video
  • 1 Supplemental Resource
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
What Will I Learn?
  • Learn how to choose the right size for sales and marketing campaigns and properly evaluate conversion rates of these campaigns.
  • Learn how to evaluate already run sales and marketing campaigns.
View Curriculum
  • Students should be familiar with basic statistics (probability, mean, variance, and the normal distribution), have some familiarity with sales or marketing campaign practices, and be willing to work with an interactive graphing worksheet.

Response based campaigns are sales or marketing campaigns where specific offers or advertisements are shown and we need to measure response rates per group. In this business problem we are choosing ad-copy and price points and are trying to pick the best combination. This is a traditional statistical problem and is often organized into what are called "A/B tests." However, more and more business professionals are required to directly plan and manage these sales and marketing campaigns without statistical or technical assistance.

This course will teach you how to plan and manage a response based marketing campaign. Learn:

  • What an A/B test is
  • How to pick campaign sizes
  • How expensive certainty can be

The method we are teaching is based on Bayesian statistical inference, so is a bit different than the more traditionally taught methods (t-tests, p-values). The advantage is the a number of natural business questions (such as how much money is at risk) are easier to encode in the Bayesian framework.

The course is unfortunately a very technical topic. To mitigate this we have wrapped the complete method in an interactive worksheet that we are sharing freely. The idea is: you supply the wisdom and judgement, the worksheet can perform all of the messy calculations.

The course structure is as follows:

  • Introduction to the course (free lecture)
  • Primer on the statistical terms needed for the course
  • Working with the worksheet
  • Review of the formal methods behind the worksheet
  • (optional) pointers to the code running the worksheet

The methods taught in this course should greatly improve your ability to plan and manage sales and marketing campaigns. You should be able to judge if this is the course for you from the free lecture.

The course largely describes a free Shiny app that performs campaign calculations for you. The app has a planning sheet and an evaluation sheet.

The purpose of the planning sheet is to use the user inputs (prior bounds on conversion rate and conversion value) to estimate an acceptable absolute error in campaign value. This acceptable error rate is used to campaign sizes that ensure the campaign chosen has a good probability of being close to the best choice in terms of relative error. A good way to use this is to enter two different rates you wish to be able to distinguish between on the planning sheet.

The evaluation sheet shows the posterior distribution of the (unknown and unobserved) true values of the campaigns conditioned on the observations the user has input.

Who is the target audience?
  • This course is for business and technical professionals who have taken a statistics or data science course, but want to learn some more of the details the business application of experiment design and evaluation.
Students Who Viewed This Course Also Viewed
Curriculum For This Course
Expand All 5 Lectures Collapse All 5 Lectures 57:32
Testing Campaign Response Rates
5 Lectures 43:32

This lecture lays out the entire course. We define what problem we are going to solve (estimating the true response rate of your sales campaigns) and how we are going to solve it (with a free interactive worksheet). The free app is linked in the lesson extras/external-resources area. Also be sure to watch the introduction video (the video that plays if you click on the image when you first load this course).

Introduction to marketing/sales campaigns and response rate testing

In this lecture we will review some common technical terms from statistics. The goal is to make these terms less of a surprise when we use them.

Primer/review of necessary statistical terms

This lecture is the meat of the course. We show how to work some problems in the campaign planning and testing worksheets. We freely share the worksheet and freely share some guide pages as downloadable materials.

Working with the campaign calculator worksheet

Try your hand at planning a sales campaign

Plan a campaign
4 questions

Now that we have worked a few exercises together we review the statistical theory behind the worksheet.

Review of the methods used in the worksheet

This optional slide deck gives pointers to the source code, references, and more.

(optional) Pointers to source code and other resources
14 pages
About the Instructor
John Mount
3.9 Average rating
151 Reviews
6,748 Students
2 Courses
Data Scientist, Win-Vector LLC

I am principal at with the data science consulting firm Win-Vector LLC. Win-Vector LLC specializes in data science research, implementation, and training. I have over 10 years of experience in research, teaching, machine learning, and data science.

I am co-author of the popular book Practical Data Science with R, and I blog often on mathematics, programming, machine learning, and optimization on the Win-Vector blog.

My profesional experience includes managing a data science group for Shopping dot com (an eBay company), working in price optimization for Rapt (acquired by Microsoft), and apply machine learning at a web-scale for Kosmix (acquired by Walmart online). My original fields of study were mathematics (AB UC Berkeley) and computer science (Ph.D. Carnegie Mellon) with a heavy emphasis on probability theory.