Hotfix Process

Khi nào hotfix

• Production critical bug (P0/P1).
• Security vulnerability (CVE).
• Data corruption risk.
• Customer-facing major break.

Hotfix flow

Bug detected → Triage → Fix branch → Code review → QA verify → Deploy → Monitor
              (15 min)  (1-4 hour)  (30 min)     (30-60 min) (15 min) (1-4 hour)

Target: bug-to-deploy < 4 hours for P0.

Branching

• Hotfix branch from production tag.
• Not from main (may have unreleased features).
• Cherry-pick fix into main after deploy.

Reduced testing scope

• Full regression: too slow.
• Focused: test the fix + affected area + smoke.
• Skip nice-to-have test, prioritize ship.

Sign-off chain compressed

• Dev fix → Lead review → QA verify → Lead deploy.
• Skip UAT (notify after).
• PM informed but not gating.

Rollback plan mandatory

• Hotfix can introduce new bug.
• Have rollback button ready.
• Monitor for 1-4h post deploy.
• Alert on error spike, latency spike.

Communication

• Incident channel (Slack #incidents) active.
• Status page update.
• Internal stakeholder notify.
• Customer notify if user-facing.

Post-hotfix

• Post-mortem within 48h.
• Add regression test.
• Update runbook.
• Why did test miss? Strategic improvement.

VN context

Banking SBV require incident notification < 2h, post-incident report < 7 days. E-com Tết hotfix common — pre-Tet rehearsal.

Anti-pattern

• Hotfix everything → quality erode.
• Skip post-mortem → repeat bug.
• Multiple parallel hotfix → conflict.