Lakehouse Architecture¶

A comprehensive guide to building unified lakehouse architectures that combine the best of data lakes and data warehouses across AWS, Azure, and Google Cloud Platform.

Table of Contents¶

1. Architecture Overview
2. Architecture Diagram Description
3. Component Breakdown
4. Table Formats
5. AWS Implementation
6. Azure Implementation
7. GCP Implementation
8. Design Considerations
9. Schema Management
10. Cost Estimation
11. Production Checklist

Architecture Overview¶

What is Lakehouse Architecture?¶

Lakehouse architecture combines the flexibility of data lakes with the performance and governance of data warehouses: - Open Storage: Data stored in open formats on object storage (S3, ADLS, GCS) - ACID Transactions: Full transactional support on data lake files - Schema Enforcement: Schema-on-write with schema evolution support - Unified Platform: Single architecture for BI, ML, streaming, and data engineering - Time Travel: Query historical data at any point in time - Performance: Indexing, caching, and query optimization on raw data

Data Lake vs Data Warehouse vs Lakehouse¶

Benefits¶

• Cost Efficiency: Cheap object storage with elastic compute
• Unified Analytics: BI, data science, and streaming on one platform
• Open Formats: Avoid vendor lock-in with open table formats
• ACID Compliance: Reliable data operations with transactional guarantees
• Time Travel: Audit, rollback, and reproduce historical queries
• Schema Evolution: Safely evolve schemas without breaking downstream consumers
• Performance: Comparable to data warehouses for structured queries

Trade-offs¶

• Complexity: More components to manage than a simple warehouse
• Maturity: Table format ecosystem is still evolving
• Tooling: Not all BI tools support lakehouse formats natively
• Optimization: Requires tuning (compaction, Z-ordering, partitioning)
• Expertise: Teams need to understand storage optimization techniques

Architecture Diagram Description¶

High-Level Architecture¶

flowchart LR
  S[Sources:<br/>logs, DBs, APIs, streams] --> ING[Ingestion]
  ING --> B[(Bronze: raw / append-only)]
  B --> SIL[(Silver: cleaned, deduped, joined)]
  SIL --> G[(Gold: aggregated / business-ready)]
  G --> SQL[SQL / BI]
  G --> ML[ML / feature store]
  G --> APP[Apps / APIs]
  META[Open table format<br/>Delta / Iceberg / Hudi] -.- B
  META -.- SIL
  META -.- G

[Data Sources] --> [Ingestion Layer] --> [Raw/Bronze Zone]
                                              |
                                     [Transformation]
                                              |
                                     [Curated/Silver Zone]
                                              |
                                     [Aggregated/Gold Zone]
                                              |
                          +-------------------+-------------------+
                          |                   |                   |
                   [BI / Reporting]    [Data Science / ML]  [Streaming Apps]

Medallion Architecture¶

The lakehouse typically uses a medallion (multi-hop) architecture:

Component Breakdown¶

Storage Layer¶

Compute Layer¶

Governance Layer¶

Table Formats¶

Comparison¶

When to Choose¶

• Delta Lake: Best if using Databricks or Spark-heavy workloads; most mature ecosystem
• Apache Iceberg: Best for multi-engine environments; strongest partition evolution and hidden partitioning
• Apache Hudi: Best for streaming upserts and change data capture (CDC) workloads

Documentation: - Delta Lake - Apache Iceberg - Apache Hudi

AWS Implementation¶

Core Services¶

Architecture Pattern¶

• S3 as the storage layer with Delta Lake or Iceberg table format
• Glue Data Catalog as the metastore
• Athena for ad-hoc SQL queries directly on S3
• Redshift Spectrum for joining warehouse and lake data
• EMR or Glue for Spark-based transformations
• Lake Formation for fine-grained access control
• Kinesis Data Firehose for streaming ingestion to S3

Documentation: - Amazon Athena - Amazon Redshift Spectrum - AWS Lake Formation - Amazon EMR

Azure Implementation¶

Core Services¶

Architecture Pattern¶

• ADLS Gen2 as the storage layer with Delta Lake format
• Azure Databricks for unified analytics (ETL, ML, SQL)
• Synapse Serverless SQL for ad-hoc queries on lake data
• Synapse Dedicated Pools for high-performance warehouse queries
• Purview for data cataloging, lineage, and governance
• Event Hubs for streaming data ingestion
• Data Factory for batch pipeline orchestration

Documentation: - Azure Synapse Analytics - Azure Databricks - ADLS Gen2 - Microsoft Purview

GCP Implementation¶

Core Services¶

Architecture Pattern¶

• GCS as the storage layer with Iceberg or Delta Lake format
• BigQuery for SQL analytics with BigLake for unified access
• BigLake tables provide a single interface to query data in GCS and BigQuery
• Dataproc for Spark-based transformations
• Dataplex for data quality, cataloging, and governance
• Pub/Sub and Dataflow for streaming ingestion
• BigQuery ML for in-place machine learning

BigLake Integration¶

BigLake is Google Cloud's lakehouse solution that unifies data lakes and warehouses: - Query data in GCS using BigQuery SQL engine - Fine-grained access control on external data - Support for Iceberg, Delta Lake, and Hudi tables - No data movement required

Documentation: - BigQuery - BigLake - Dataplex - Dataproc

Design Considerations¶

Partitioning Strategy¶

• Time-based partitioning: Most common; partition by date, hour, or month
• Column-based partitioning: Partition by high-cardinality columns (region, customer_id)
• Hidden partitioning (Iceberg): Users query without knowing partition structure
• Over-partitioning risk: Too many small files degrades performance
• Rule of thumb: Aim for partition sizes of 100 MB to 1 GB

File Size Optimization¶

• Target file size: 128 MB to 1 GB for Parquet files
• Small file problem: Too many small files causes slow queries and high metadata overhead
• Compaction: Run regular compaction jobs to merge small files
• Auto-optimization: Delta Lake auto-optimize, Iceberg automatic compaction

Query Performance¶

• Z-Ordering / Clustering: Co-locate related data for faster range queries
• Column statistics: Maintain min/max statistics for data skipping
• Caching: Use SSD caching or query result caching for frequent queries
• Materialized views: Pre-compute common aggregations

Schema Management¶

Schema Enforcement¶

• Schema-on-write: Reject data that does not match the expected schema
• Schema validation: Validate at ingestion time to catch issues early
• Data type enforcement: Ensure correct data types for all columns
• Not-null constraints: Enforce required fields

Schema Evolution¶

Best Practices¶

• Backward compatibility: Adding columns is safe; removing columns requires coordination
• Versioned schemas: Track schema changes in version control
• Default values: Use default values for new columns to avoid null issues
• Schema registry: Use a schema registry for streaming data (Avro, Protobuf)
• Testing: Validate schema changes against downstream consumers before deploying

Time Travel and Versioning¶

• Snapshot queries: Query data as it existed at a specific point in time
• Rollback: Restore a table to a previous version if errors are detected
• Audit: Compare current data with historical versions for auditing
• Retention: Configure how long historical versions are retained (storage cost trade-off)
• Reproducibility: Re-run ML experiments against the exact same data

-- Delta Lake time travel
SELECT * FROM my_table VERSION AS OF 5;
SELECT * FROM my_table TIMESTAMP AS OF '2024-01-01';

-- Iceberg time travel
SELECT * FROM my_table FOR SYSTEM_TIME AS OF TIMESTAMP '2024-01-01 00:00:00';

-- Rollback
RESTORE TABLE my_table TO VERSION AS OF 3;

Cost Estimation¶

AWS (Monthly Estimate - 10 TB Lakehouse)¶

Azure (Monthly Estimate - 10 TB Lakehouse)¶

GCP (Monthly Estimate - 10 TB Lakehouse)¶

Production Checklist¶

Storage and Format¶

• Table format selected (Delta Lake, Iceberg, or Hudi)
• Medallion architecture zones defined (bronze, silver, gold)
• Partitioning strategy designed and documented
• File size targets configured (128 MB - 1 GB)
• Compaction jobs scheduled
• Storage lifecycle policies configured for cost optimization

Data Quality¶

• Schema enforcement enabled on all tables
• Data quality rules defined for each zone
• Quality monitoring dashboards operational
• Alerting configured for quality violations
• Dead letter handling for rejected records

Performance¶

• Z-ordering or clustering configured for frequently queried columns
• Column statistics maintained and up to date
• Query performance benchmarks established
• Caching strategy implemented for hot queries
• Small file compaction automated

Governance and Security¶

• Access control implemented (table, column, row level)
• Data catalog populated with business metadata
• Data lineage tracking configured
• Audit logging enabled for all data access
• Encryption at rest and in transit configured
• PII detection and masking implemented

Operations¶

• Pipeline monitoring and alerting configured
• Backup and recovery procedures tested
• Time travel retention policies configured
• Cost monitoring and optimization in place
• Schema evolution procedures documented
• Runbooks for common operational tasks created

Related Guides: - Data Pipeline ETL Architecture - Data Mesh Architecture - Service Comparison - Databases