A/B Testing & Experimentation

Technical

A/B testing và experimentation: hypothesis, metrics, guardrails, sample planning, QA tracking, analysis, decision rules và pitfalls.

Roadmap — Cách học và đạt kỹ năng

A/B Testing & Experimentation là gì?

A/B testing là cách ra quyết định bằng controlled experiment. Nó không phải dashboard decoration. Một experiment tốt bắt đầu bằng hypothesis rõ, metric chính, guardrail metrics, sample plan, QA tracking và decision rule.

A/B testing lifecycle from hypothesis to decision — Experiment lifecycle: hypothesis, metrics, design, run, analyze, decide. Không đổi metric giữa đường.

Experiment plan template

Field	What to write	Example
Hypothesis	If we change X, Y will improve because Z.	If checkout shows delivery date earlier, conversion improves because uncertainty drops.
Primary metric	Main decision metric.	Checkout completion rate.
Guardrails	Metrics that must not get worse.	Refund rate, support tickets, payment failures.
Audience	Who is eligible/excluded.	New mobile users, excluding COD orders.
MDE/sample	Minimum detectable effect and traffic need.	Detect +3% relative lift, 14 days.
Decision rule	Ship/iterate/stop condition.	Ship if primary improves and guardrails neutral.

Case study: pricing page CTA

Team tests “Start free trial” vs “Book demo”. Primary metric: qualified signup. Guardrail: sales-qualified lead quality. Result: free trial increases signup but lowers SQL quality for enterprise traffic. Decision: segment CTA by company size instead of global winner.