Post-Mortem

Định nghĩa

Structured review of incident — identify what happened, why, how to prevent.

Blameless culture

• Focus on system/process failure, not individual.
• Anyone can ask "5 whys" without fear.
• Mistake = learning opportunity.

Template

1. Incident summary (5 lines max)
Impact (users affected, revenue lost, duration)
Timeline (UTC, who did what when)
Root cause analysis (5 whys)
What went well
What went wrong
Action items (owner + deadline)
Lessons learned (for org)

Run a post-mortem meeting

• Schedule within 48h of incident.
• 60-90 phút.
• Invite: on-call eng, manager, QA, PM, customer-facing.
• Note-taker dedicated (not facilitator).
• Action items go to JIRA same day.

Action item categories

• Prevent: code fix, test added.
• Detect: alert added, dashboard.
• Mitigate: runbook, automation.
• Process: training, doc update.

Common failures

• Skip post-mortem khi "small" incident → repeat.
• Action items không follow up.
• Blame culture → people hide incident.
• No publish → org doesn't learn.

QA role in post-mortem

• Identify why test miss this.
• Add regression test.
• Update test strategy if pattern.
• Share with QA org.

Etsy / Google practice

• Post-mortem public to entire eng org.
• Quarterly summary "lessons learned" newsletter.

VN context

Banking SBV regulation: incident report mandatory within 24h, root cause within 7 days.