Data Contracts

Problem

Source team change schema → break downstream consumer → fire drill.

Concept

Formal agreement between producer + consumer on schema, freshness, quality.

Contract content

• Schema (column name, type, nullable).
• Cardinality (how many row expected).
• Freshness (lag from event to warehouse).
• Quality SLO (completeness %).
• Breaking change protocol.
• Versioning.

Tools

• Schema Registry — Confluent (Kafka).
• Protobuf / Avro — strong typing.
• dbt contracts — built-in 2024.
• Custom JSON schema + CI validation.

Workflow

• Producer propose change.
• Impact analysis (lineage).
• Consumer notify + grace period.
• Both ship simultaneously (versioned).
• Migrate consumer.
• Deprecate old.

Cultural shift

• Source data team accountable to consumer.
• "Data product" mindset.
• Consumer can demand quality.

Anti-pattern

• Schema change "silent" → break overnight.
• No versioning → can't roll back.
• No consumer awareness → producer change without consult.

ROI

• Reduce data incident 50%+.
• Faster consumer onboarding.
• Trust restored — analyst can rely on data.

Implementation tip

• Start with most critical 5-10 datasets.
• Use existing CI/CD — no need new platform.
• Iterate based on incident postmortem.