Buying for a Team? Gift This Course
Wishlisted Wishlist

Please confirm that you want to add Build a Testing Program: CRO, MVT, & A/B Testing Theory to your Wishlist.

Add to Wishlist

Build a Testing Program: CRO, MVT, & A/B Testing Theory

Increase conversions, higher ROI, and more profits by mastering conversion rate optimization strategies and theories.
4.0 (15 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
885 students enrolled
Created by Rhett Norton
Last updated 7/2014
$10 $20 50% off
4 days left at this price!
30-Day Money-Back Guarantee
  • 4 hours on-demand video
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Have a coupon?

COURSE UPDATED ON: Jan 8, 2014 - added new lectures on documentation infrastructure

These principles work. My last test with a client saw a 30% increase in their conversion rate. You can have the same results by learning these theories.

Over 600 Student and growing.

Over 4 hours of video content. Plus the video is interesting and valuable. :-)

***Disclaimer: I am building these courses for my own site and I make several references to other courses even though they aren't done yet. I am not trying to upsell anything since you have all my produced content, I simply recorded these videos with the end listener in mind which would be someone from my site. In the mean time, you can enjoy these courses at a nice discount.

Having worked at Omniture/Adobe as a consultant for over 6 years I have seen hundreds of companies fail and succeed with testing. I have seen thousands of tests and I have seen the patterns for success from the best and worst tests. If you are ready to move into the data driven world of testing and you want to learn how to do it right, this course is for you.

Learn what testing is and how testing theory can help your business. In this course, I will introduce you to what it means to grow you testing program as you mature in testing. I will also introduce you the three pillars that every successful testing company needs to understand as they learn more advanced testing theories. Much more than conversion rate optimization, this strategy course will give you a framework for knowing how to maximize your success with your testing efforts.

Course Includes:

  • Overview of testing and what it is
  • How to choose the best metrics for testing that focus on ROI
  • How the theories and strategies of testing can make you more money
  • Calling winners and avoiding false positives
  • Using 8 simple steps to improve your testing
  • Knowing where to begin with your test ideas
  • Minimizing your risk and understanding risk
  • Building infrastructure for your testing program
  • Much, much, more.

For you reference, I will be adding more topics to the course as I complete them over time. Some topics I have planned are below.

  • Executive Involvement - Sponsorship and Shielding
  • Other Key Players for your Testing Program
  • Creating Buyin
  • Adoption and Usage
  • Building Your Testing Program Over Time
  • Ideation - How to come up with awesome test ideas
  • Ongoing Disciplined Iteration
  • Efficiency and Testing
  • Testing and Your Analytics Data
  • Multiplying Learnings
  • Avoiding Neutral/Flat Tests
  • Monetization - Annualized Impact of the Testing Program
  • Speed of Testing
  • Jumping to Conclusions
  • Governance Models
  • Creating a Culture of Testing
  • Involvement Across Business Units
  • Getting Outcomes vs. Influencing Outcomes
  • Forecasting
  • Influencing Your Organization for Success
  • Elements Methodology
  • Marketing Methodology
  • Multivariate Strategy
  • Segmentation Strategy
  • Personalization Strategy
  • Testing Mindset vs. Analytics Mindset
  • Understanding Variance/Noise and Running a Variance Study
  • Statistics of Testing
  • Psychology and Testing
  • Assumptions, Biases, Dependent Logic, and other Logical Fallacies
  • Recommendations and Testing
  • The Market Influence, Seasonality and Holiday Testings
  • Testing Velocity
  • Sharing Knowledge
  • Rules of Action
Who is the target audience?
  • Conversion rate optimizers
  • Digital Marketers - Optimization and Analytics Professionals
  • Anyone who wants to learn how to increase profits for their online buiness
  • Those wanting a data driven approach to making more money
Students Who Viewed This Course Also Viewed
What Will I Learn?
Increase your conversion rates by starting a testing program
Maximize your profits through the discipline of testing and learning
Help you understand what it means to use Theory in testing
Give you some foundational strategic theory to help you excel in your testing efforts
View Curriculum
  • Desire to be the best tester, make more profits, and learn about your site and visitor behavior
  • Come ready to learn theories that can apply to any site
Curriculum For This Course
Expand All 22 Lectures Collapse All 22 Lectures 04:13:17
Introduction to Testing
2 Lectures 15:04

Problem- Companies that don't know about testing or that don't do testing are missing out on learning what works for their customers and what makes them money as a business.

Solution- Learn what it means to do online testing. Discover the value of learning what your customers truly want through digital optimization. Maximize your profits with power of running tests on your digital properties.

Preview 05:59

Problem - Some proclaimed experts like to say that as you become more advanced with testing you will run more advanced tests - referring to the type of tests. Like you will only run Multivariate tests
Solution - Types of tests are like tools in a tool box and learning what all the testing tools are and when to use them will help you choose the type of test that you should run.

Preview 09:05
Strategic Theory and Digital Testing
3 Lectures 25:43

Problem- Companies put bad test ideas into their testing tools and get results that are meaningless and in some cases misleading.
Solution- This lecture is an introductory lecture that highlights the importance of putting good inputs into the testing program. This lecture will be good background for the more advanced and in depth courses that follow.

Preview 05:30

Discover the applicability of testing theory to your business. By using theory you can learn the strategies developed by seeing thousands of tests across hundreds of companies.

Problem - Companies don’t understand what theory is and why it is so important.
Solution - I will help you see why testing theory is vital to your testing business.

Preview 05:37

Problem- Many companies fear making changes with testing because they are worried that their changes will have a negative impact on conversion. They fear risk and think that big changes are even riskier.

Solution- In this course we will cover how risk is both positive and negative. We will review the implications of risk and help you understand how risk can work to your advantage testing. We will also cover the two best strategies for hedging against negative risk.

  • Real Risks of Testing
  • Switching it up on the Customers

  • Positive and Negative Risks

  • Intelligent Risk Taking

  • Turning Negative Risk into Positive Risk

  • Summary and Maturity Audit

Risks and Testing
Foundational Strategic Theories for a Testing Program
8 Lectures 01:31:02

Problem- Too many testers don't give proper respect to testing as a discipline and they neglect the fact that proper testing requires both art and science. This can lead to very haphazard testing and causes companies to fail in their testing efforts. Companies think that a testing program is about running tests, but it is much much more than that.

Solution- In this course we will review how the scientific method of testing can give focus to your testing efforts. This will be a foundational course that will lead into more advanced methodology and strategic theory courses. This course will also illustrate why some of the foundational infrastructure and process need to be in place.

Scientific Method of Testing Part 1

  • Define a question
  • Gather information and resources (observe)
  • Form an explanatory hypothesis
  • Test the hypothesis by performing an experiment and collecting data in a reproducible manner
  • Analyze the data
  • Interpret the data and draw conclusions that serve as a starting point for new hypothesis
  • Publish results
  • Retest
Scientific Method of Testing Part 2

Problem- Too many companies have their pet ideas they have been waiting to test and more often than not those random ideas aren't the best tests they could run to learn and maximize efficiency. When they actually test a few of their ideas they get flat or losing results and then they get frustrated and the testing program looses steam.
Solution- The first test of your site or a new test of series doesn't have to be a wild guess. It can be the most disciplined and efficient of all your tests. Your first test can be the beginning of a journey that can lead you to the buried treasure of your site. In this course I will show you the best questions to ask that will help you learn the most, maximize the value of your initial tests, and lead to the best follow up tests.

  • Off by a few degrees
  • First Question
  • Corporate Home Page Sickness
  • Landing Page Test and Existence of Pages
  • Testing Flow Pages
Beginning a Testing Series and Existence Testing- Part 1

  • Benefits
  • Execution and Using Data

  • Iterations

  • Second Question - Existence of Elements

  • Summary and Maturity Audit

Beginning a Testing Series and Existence Testing- Part 2

Problem- Companies have poor testing when they don’t properly formulate a good testing question. Most think that a hypothesis is a good thing but they are actually hurting themselves by doing it improperly. Testers avoid seeing plausible alternatives because of they want to prove themselves right. Sometimes companies will excel at creating good questions but in doing so they lose sight of their ultimate goal because they don’t have the right measurement.

Solution- In this course we will discuss what it means to formulate good testing questions. We will define congruence bias, help you recognize when you are stuck in it, and give you some pointers on how to avoid this bias in your testing to create better tests and make more money.

Asking the Right Questions and Avoiding Congruence Bias - Part 1

Learn the 7 critical things that will help you learn how to recognize and avoid congruence bias in your testing efforts

Asking the Right Questions and Avoiding Congruence Bias - Part 2

Problem- Through ignorance or the unrestrained zeal of wanting to do more, many companies invalidate their testing results by taking actions that make their data dirty and create uncertainty.

Solution- In this course we will review how you can keep your results pure and undiluted so that when you call a winner you can be sure it is truly a winner. We will explore what it takes to avoid dirtying your data. We will debunk the common excuses companies use to not take these highly strategic actions.

Dirty Dash Testing or Keeping Your Testing Data Clean-Part1

  • The Mud Traps of Multiple Changes - how to recognize and avoid them
  • Excuses for Wallowing in the Mud
  • Summary and Maturity Audit
Dirty Dash Testing or Keeping Your Testing Data Clean-Part 2
Foundational Infrastructure and Execution for a Testing Program
7 Lectures 01:21:55

In this lecture we will discuss what it takes to build a solid testing program. I will introduce you to the three pillars of every successful testing program. I will also show you the stages of evolution that most programs experience.

Problem- Many companies think they should just be running tests and don’t realize the power of building a testing program. Other companies want a testing program, but they don’t know what it takes to develop one. Not having a program, or having a program that isn’t built properly severely limits the gains that can come through testing.
Solution- In this course we will discuss what it takes to build a solid testing program. I will introduce you to the three pillars of every successful testing program. I will also show you the stages of evolution that most programs experience.

Preview 09:10

Problem- Companies regularly call winners that aren't winners and in follow up tests they are surprised to see a winning experience become the loser. The majority of companies incorrectly rely on one measure to interpret results and don't take into the account the other measures for calling a winning experience. When you call an incorrect winner you end up hurting your company by pushing losing experiences. Additionally, Many companies will find a winning experience and then not push that experience live.

Solution- In this course I will show you the measures you should look at to declare a winner and what it looks like to have a balance as you use both metrics and the human element to find and declare a winning experience. I will give you guidance so you don't declare an experience the winner when it actually isn't. I will also give you some strategies for making sure your winning experiences get pushed live.

Calling a Winner - Metrics and the Human Element - Part 1

Framework for Calling Winning Experiences

1. Primary metric tied to revenue
2. Sufficient data
3. Consistent data
4. Differentiated data
5. Statistical Confidence (NICE TO HAVE)

Calling a Winner - Metrics and the Human Element - Part 2

Spontaneous Generation and Process

Advantages of a Good Process

1. Define all the activities

2. Define who will do each activity

3. Define the sequence

4. Define notification mechanisms and hand-offs for each stage

5. Create accountability

6. Fine tune the process as needed - hire new resources, the program grows

Creating Infrastructure with Process - Part 1

1. Define all the activities
2. Define who will do each activity
3. Define the sequence
4. Define notification mechanisms and hand-offs for each stage
5. Create accountability

6. Fine tune the process as needed - hire new resources, the program grows

Creating Infrastructure with Process - Part 2

In this course we will cover the eight essential documents every testing program needs and I will teach you the key principles behind each document so you can use documentation as a strategic advantage in your testing program.

Creating Infrastructure with Documentation - Part 1

Creating Infrastructure with Documentation Part 2
Foundational Organization and Leadership for a Testing Program
2 Lectures 39:33
  • Professional Substitutions
  • Metrics in Analytics
  • Metrics in Testing
  • Compare and Contrast
  • Primary Metric
  • Profitability and your Primary Metric
  • What is the Job of the website?
  • What should your primary metric be?
  • Advanced Scoring
  • Call a winner with primary metric
  • Parable of the primary metric
  • Choosing Secondary Metrics
  • Using Secondary Metrics
  • Metrics and your company
  • Summary and Maturity Audit
Metrics to Money - Choosing and Using Primary and Secondary Metrics

Problem- Teams that aren’t trained well aren’t as successful. Companies that don’t invest in training their testing teams don’t maximize their potential. Turnover is one of the biggest challenges for companies and testing programs and on boarding programs to train resources is often non-existent in many organizations. The new employees are expected to learn on the job and the school of hard knocks isn’t as profitable for the company or efficient for the employee.
Solution- In this course we will address the importance of training the core testing team. We will discuss the importance of expanding training outside the testing team and the positive impact that can have of on culture. In this course we will discuss strategies to minimize the impact of turnover and losing expertise. We will also review the way to get maximum value out of this training course as you ramp up new employees.

  • Ignorance and it's impact
  • Herzberg's model
  • Combating the effects of people moving
  • Creating Center of Excellence
  • Internal Knowledge Expansion
  • Overcoming Ignorance with Enablement and Training
  • Skill Specific Training
  • Summary and Maturity
Turnover, Training, Onboarding, and Enablement
About the Instructor
4.0 Average rating
15 Reviews
885 Students
1 Course
Expert in Testing Theory and Digital Optimization

For more than 6 years, Rhett was a consultant at Adobe/Omniture where he consulted for hundreds of companies. Rhett led the Target consulting team at Adobe where he grew the testing and targeting practice. He has consulted for and trained other consulting agencies including Accenture, RazorFish, and Deloitte. He has personally seen and developed thousands of tests and tested across every major industry with the biggest and best online companies.

A few of the clients Rhett has consulted for include: Amazon, Staples, Citi, Kohls, Fidelity, Redbox, T-Mobile, Delta, Adobe, Under Armour, Ford, Lane Bryant, Polo, Sears, Home Depot, Lego, Wal-Mart, Avon, Walgreens, American Eagle, Nordstroms, VideoJet, Chegg,, Safeway, Williams-Sonoma, Virgin Mobile, Kelley Blue Book, Scholastics, Norwegian Cruise Lines, and many more.

Rhett Norton is a natural leader that is ever challenging companies and individuals to create and reach higher levels of strategic thinking and monetizable success. He is passionate about online optimization strategy and influencing the culture and mindset of the organizations that he works with. He has had a dramatic positive impact on the best digital marketing companies as he has helped them build their testing organizations and prove tangible ROI through revenue driven A/B testing and multivariate testing strategies.

Rhett is a natural teacher and enjoys the challenge and opportunity of helping people understand the optimal way of doing things. He likes to influence the way that people think so they are empowered to think for themselves. In his teaching, Rhett is a master of using analogies, stories, and real life examples to bring complex topics into simple clear views. He has the ability to distill large amounts of information through easily digestible material.

Rhett is always focused on driving success by focusing on the efficiency of each action and challenging the biases and assumptions that impede success. He understands what complex organizations need to succeed and knows how to help them with governance, infrastructure, and execution. He is talented in teaching and can distill complex testing methodologies into simple actions that any company can take. He helps companies work towards best in class optimization and yet helps them to grow where they are currently.

Rhett received an MBA from Brigham Young University. He also holds a Bachelor of Science in Business Management from BYU.

Report Abuse