A/B Testing in Python

Learn to define and measure metrics for a digital product using a fictional A/B test with Kit and Gram, including engagement, revenue, and performance KPIs.

Explore practical A/B testing calculations using two data sets to measure daily active users and click-through rate, comparing control and test groups for sponsored posts.

Master data instrumentation and tracking to collect, clean, and aggregate user events over eight days of metrics, and fuel dashboards for A-B testing, computing daily activity and ad click-through rate.

Explore calculating metrics from raw datasets using Python in Jupyter notebooks to compute daily active users and click-through rate, assess distribution, and discuss data quality and hypotheses for AB testing.

Define the experiment design by selecting testable hypotheses and metrics. Assess when an idea is worth testing, considering minimum detectable error, power, significance, and non-inferiority margins.

Calculate sample size and test duration for an A/B test by balancing minimum detectable effect, power, and business constraints to determine exposure and group sizes for reliable results.

Define statistical significance and alpha, assess false positives and type i/ii errors, and understand test power while planning sample size and one-tailed vs two-tailed tests for a/b testing in Python.

Learn how to calculate A/B test sample sizes with z tests, alpha and beta, and one- or two-tailed options for binomial and continuous metrics.

Calculate statistical significance and A/B test sample sizes in Python, using alpha, beta, minimum detectable effect, and pooled proportion for binary and continuous metrics.

Define hypotheses and success metrics, assign users randomly, and monitor with predefined significance and power. Analyze results for pre-test bias and time trends, then share findings with stakeholders.

Present a mock A/B test results deck, outlining feature, hypotheses, metrics, duration, group size, and assumptions; explain metric calculations, show significance, and propose future improvements.

Outlines A/B test analysis process: assign users to test and control, check pre-test bias, evaluate UX and business metrics, compute significance and non-inferiority, and present results with a Python notebook.

Compare activity and click-through rate between groups using t tests to show pre-test similarity and post-test differences, demonstrating significant improvements in an a/b test.

Explore advanced a/b testing considerations, including risk assessment, multiple concurrent tests, cross-market features, and privacy and ethics guidelines aligned with ACM and GDPR.

Differentiate anonymized, anonymous, and pseudonymous data and assess how data linking and geographic pricing raise privacy, non-discrimination, and ethics concerns in A/B testing.

Explore advanced interview questions for product data scientists after presenting A/B test findings and a take-home assignment, discuss potential improvements, and consider further learnings from testing features differently.

Explore interview scenarios for an A/B test analysis, highlighting independence violations, seasonality, and cluster-based randomized designs such as network clustering and ego cluster randomization to ensure valid rollout.

Examine how new iOS feature exposure dilutes A/B test results and learn to assign users by OS version to measure non-dilutive impact.

Explore novelty and primacy effects in A/B testing, emphasizing long-term impact, time-based events, and tailored tests for new versus existing users with qualitative insights.

Learn how multiple testing in A/B experiments affects false positives and how bonferroni correction or false discovery rate mitigates risk, with examples of testing multiple button colors.

Discover strategies for when you cannot AB test, using qualitative research, surveys, and interviews to understand user behavior, and apply generalized synthetic control group and time series causal impact analysis.

Clarify metrics, ask questions, and explain your decisions and assumptions during a B test interview. Align your hard skills with business needs by mapping stakeholders and KPIs.

Master designing, analyzing, and presenting A/B test results to stakeholders, anchored in statistics, SQL, and Python or R, with a practical, business-focused mindset.