Chaos Engineering

Definition

Intentionally inject failure to verify system resilience.

Netflix origin

Chaos Monkey — random terminate EC2 instances daily. Force engineers build fault-tolerant.

Principles (Principles of Chaos)

• Define steady state.
• Hypothesize state remains.
• Inject real-world events.
• Disprove hypothesis if state breaks.
• Run in production (carefully).

Failure modes to test

• Server crash — instance terminate.
• Network partition — region disconnect.
• High latency — add delay 5s.
• Disk full.
• DNS failure.
• Dependency down — DB unavailable.
• Region failure — entire AZ.

Tools

• Chaos Monkey (Netflix).
• Chaos Mesh (CNCF, Kubernetes).
• Gremlin — commercial.
• Litmus — Kubernetes.
• AWS Fault Injection Simulator.

Game day

Quarterly drill:

• Pick scenario (e.g., DB primary fail).
• Run in staging first.
• Observation: how long detect? mitigation work?
• Improvement: alerting, runbook, automation.

Pre-requisite

• Strong monitoring + alerting.
• On-call rotation.
• Rollback automation.
• Stakeholder buy-in.

Don't start chaos without

• Tier-1 services SLO defined.
• On-call team trained.
• Mature post-mortem culture.

QA role

• Design chaos scenarios.
• Run with SRE.
• Capture lessons learned.
• Add resilience test to regression.

VN context

Banking VN slowly adopting (regulatory caution). Tiki, Shopee VN have game day quarterly. SRE+QE collaboration.