
Discover how to build a successful a/b testing and experimentation program by testing the right things, shaping the right team and culture, and solving the most important user problems.
A/B testing compares two or more variants shown at random to measure performance toward revenue growth using statistical analysis; it highlights building right process, team, and culture to scale experiments.
Begin with the experimentation lab by gathering data, then prioritize problems, form hypotheses, design solutions, build and test them, and learn to improve.
Gather data early to understand how people use your product and where they get stuck, using pathway analysis and final analysis to map traffic, landing pages, and conversion paths.
Prioritize user problems to guide experiments and avoid wrong solutions. Use open-ended surveys and bucketed categories to uncover why users struggle and size the impact for focused improvements.
Form testable hypotheses by linking the problem to evidence and clearly defining success metrics. Use a template to articulate the expected changes in KPI and prioritize hypotheses for next steps.
Prioritize product hypotheses by scoring impact, confidence, and ease with ICE or a custom framework, then select top ideas for quick prototypes or full experiments, validated by target users.
Learn from experiments by analyzing results, validating problem solving, and tracking KPIs and dashboards. Share quarterly learnings, apply the prioritization framework, and accelerate the experimentation loop to deepen customer understanding.
Assemble a cross-functional team with an experiment owner, engineer, and user experience designer, and establish clear decision makers to manage dependencies and accelerate experimentation.
Discover the key qualities of a good experimenter for product teams, including curiosity, analytical thinking, comfort with uncertainty, attention to detail, and leadership in driving A/B testing.
Foster a culture that embraces experimentation and learning in your A/B testing program. Reward wins and failed experiments, acknowledge eight out of ten fail, and shift to learning.
Finish the online course and join the 10 percent club by completing the a/b testing crash course for product managers. Reflect on your growth and share what helped.
Learn how to build successful experiments to improve your product and grow your business from Udemy's Senior Director of Product.
Many product experiments fail because we test the wrong things. If you are on a budget and have limited resources, like most of us out there, it's critical that you spend your time the right way. If you just started to think about running A/B tests, it could be overwhelming to figure out where to start and what to test. In this course, I will show you the step by step process we use at Udemy to figure out what to experiment with. Also you will learn more about the roles in an experiment team, how to manage dependencies, most important qualities of a strong experiment owner and how to hire the right people (includes interview questions for hiring experiment owners). And finally, you will learn how to think about the team culture to make your experimentations successful.
This is a short and sweet course that aims to give you advanced frameworks and ideas to take your product to the next level by running A/B tests. I can't wait to see you inside!
Note: In this course, we don't cover A/B testing tools or A/B testing statistics.