Data Platform Architecture

4 patterns

• Data Warehouse — structured, BI-focused. Snowflake/BigQuery/Redshift.
• Data Lake — raw, multi-format. S3/GCS + Hive metastore.
• Lakehouse — combine. Databricks Delta, Iceberg, BigLake.
• Data Mesh — domain-distributed.

Layered architecture

• Bronze — raw ingest.
• Silver — cleaned, validated.
• Gold — business-ready, aggregated.

Components

• Ingestion (Fivetran, Airbyte).
• Storage (warehouse / lake).
• Transformation (dbt, Spark).
• Orchestration (Airflow, Dagster).
• Quality (Great Expectations, Monte Carlo).
• Catalog (Atlan, DataHub).
• BI (Looker, Power BI).
• Activation (Hightouch, Census).
• ML (Databricks ML, Vertex AI).

Decision factors

• Volume: < 100GB → simple warehouse OK.
• Variety: lots of unstructured → lake/lakehouse.
• Velocity: real-time → streaming layer.
• Veracity: high quality bar → strong governance.
• Value: clear use case prioritize.

Anti-pattern

• "Data lake of mud" — dump everything, no governance.
• Over-engineer for scale not needed.
• Tool sprawl — 5 ingestion tools.
• No clear ownership.

VN

• Most start with BigQuery + dbt — simple, scalable.
• Lakehouse (Databricks) growing in 2024+ for ML use case.