Data Observability

Concept

Equivalent of "DevOps observability" — for data. Detect + diagnose data issue automatically.

5 pillars (Monte Carlo)

• Freshness — data up-to-date?
• Volume — row count normal?
• Schema — schema unchanged?
• Distribution — values within expected range?
• Lineage — what breaks downstream?

Tools

• Monte Carlo — leader, AI-based anomaly detection.
• Bigeye — similar, open-source roots.
• Soda — OSS, SQL-based test.
• Anomalo — ML-driven.
• Great Expectations — assertion framework.

SLA mindset

• Data SLO per dataset: P99 freshness, completeness.
• Error budget — allow some incident before action.
• Postmortem culture.

Implementation

• Identify critical dataset (top 20).
• Define SLA per dataset.
• Instrument observability tool.
• Alert + runbook.
• Quarterly SLA review.

Anti-pattern

• Alert fatigue — too many false positive.
• Monitor everything → cost + noise.
• No owner — alert goes unanswered.

Org pattern

• Data on-call rotation.
• Incident channel in Slack.
• Public incident page for stakeholder.

VN adoption

• Early stage — most companies "find out from user".
• Monte Carlo / Bigeye in top tech companies 2024+.
• OSS alternative for budget-tight.