Data Mesh Architecture¶

A comprehensive guide to implementing domain-oriented, decentralized data architectures across AWS, Azure, and Google Cloud Platform.

Table of Contents¶

1. Architecture Overview
2. Architecture Diagram Description
3. Component Breakdown
4. AWS Implementation
5. Azure Implementation
6. GCP Implementation
7. Design Considerations
8. Data as a Product
9. Federated Governance
10. Cost Estimation
11. Production Checklist

Architecture Overview¶

What is Data Mesh?¶

Data mesh is a decentralized data architecture that organizes data by business domain: - Domain Ownership: Each business domain owns and manages its own data - Data as a Product: Data is treated as a product with defined quality, SLAs, and discoverability - Self-Serve Data Platform: Infrastructure provides a platform for domains to publish and consume data - Federated Computational Governance: Governance is automated and embedded, not centralized - Decentralized Architecture: No central data team owns all data pipelines

Four Pillars of Data Mesh¶

1. Domain-Oriented Ownership: Data is owned by the teams that understand the business domain
2. Data as a Product: Each data product has an owner, SLAs, documentation, and quality guarantees
3. Self-Serve Data Infrastructure: A platform that makes it easy for domains to create and manage data products
4. Federated Computational Governance: Global policies enforced through automation, not manual review

Benefits¶

• Scalability: Data ownership scales with organizational growth
• Domain Expertise: Teams closest to the data manage it
• Reduced Bottlenecks: No central data team as a bottleneck
• Faster Time to Value: Domains can independently create data products
• Quality: Domain experts ensure data accuracy and relevance
• Innovation: Teams can experiment without cross-team dependencies

Trade-offs¶

• Organizational Change: Requires significant cultural and organizational shift
• Duplication: Some data may be duplicated across domains
• Standardization: Balancing domain autonomy with interoperability
• Skill Requirements: Each domain needs data engineering capabilities
• Cost: Decentralized infrastructure can be more expensive than centralized
• Governance Complexity: Federated governance is harder to implement than centralized

Use Cases¶

• Large enterprises with multiple business units and data domains
• Organizations where a central data team has become a bottleneck
• Companies adopting microservices and domain-driven design
• Industries with complex data regulations requiring domain accountability
• Organizations needing to scale their data analytics capabilities

When to Avoid Data Mesh¶

• Small organizations with a single data team
• Simple data landscapes with few domains
• Organizations without mature engineering practices
• Teams not ready for cultural shift to domain ownership

Architecture Diagram Description¶

High-Level Architecture¶

flowchart TB
  subgraph SD[Sales domain]
    SS[Source systems] --> SI[Ingestion] --> SP[Data product:<br/>orders, revenue]
  end
  subgraph PD[Product domain]
    PS[Source systems] --> PI[Ingestion] --> PP[Data product:<br/>catalog, inventory]
  end
  subgraph CD[Customer domain]
    CS[Source systems] --> CI[Ingestion] --> CP[Data product:<br/>customer 360, segments]
  end
  SP --> CAT[Federated catalog<br/>+ governance]
  PP --> CAT
  CP --> CAT
  CAT --> CON[Cross-domain consumers:<br/>analytics, ML, BI]

+------------------+    +------------------+    +------------------+
|   Sales Domain   |    |  Product Domain  |    | Customer Domain  |
|                  |    |                  |    |                  |
| [Source Systems]  |    | [Source Systems]  |    | [Source Systems]  |
|       |          |    |       |          |    |       |          |
| [Ingestion]      |    | [Ingestion]      |    | [Ingestion]      |
|       |          |    |       |          |    |       |          |
| [Data Product]   |    | [Data Product]   |    | [Data Product]   |
|  - Sales Orders  |    |  - Product Cat.  |    |  - Customer 360  |
|  - Revenue       |    |  - Inventory     |    |  - Segments      |
+--------+---------+    +--------+---------+    +--------+---------+
         |                       |                       |
         +----------+------------+-----------+-----------+
                    |                        |
          [Data Product Catalog]    [Self-Serve Platform]
                    |                        |
          [Federated Governance]    [Monitoring / Quality]

Data Product Structure¶

Each data product consists of: 1. Input Ports: Data ingestion from source systems 2. Transformation: Domain-specific business logic 3. Output Ports: APIs, datasets, events for consumers 4. Metadata: Schema, documentation, lineage, quality metrics 5. Observability: Monitoring, alerting, SLA tracking

Component Breakdown¶

Platform Components¶

Data Product Components¶

AWS Implementation¶

Core Services¶

Multi-Account Strategy¶

+---------------------------+
|   Management Account      |
|   - AWS Organizations     |
|   - Central Governance    |
+---------------------------+
         |
    +----+----+----+
    |         |         |
+---------+ +---------+ +---------+
| Sales   | | Product | | Customer|
| Account | | Account | | Account |
| - S3    | | - S3    | | - S3    |
| - Glue  | | - Glue  | | - Glue  |
| - Lake  | | - Lake  | | - Lake  |
| Format. | | Format. | | Format. |
+---------+ +---------+ +---------+

Implementation Pattern¶

• Each domain gets its own AWS account via AWS Organizations
• Lake Formation manages cross-account data sharing
• Glue Data Catalog provides federated metadata
• S3 buckets store domain data products
• IAM policies enforce access control per data product
• EventBridge enables cross-domain event notifications

Documentation: - AWS Lake Formation - AWS Glue - Amazon S3 - AWS Glue Data Quality

Azure Implementation¶

Core Services¶

Subscription Strategy¶

• Each domain gets its own Azure subscription or resource group
• Purview provides cross-subscription data catalog and governance
• ADLS Gen2 containers organized by data product
• Synapse workspaces per domain for independent processing
• Azure Policy enforces tagging and compliance standards
• Managed identities for cross-domain access

Key Integration Points¶

• Purview scans data sources across domains automatically
• Data Share enables snapshot and in-place sharing
• Synapse Link provides near-real-time analytics from operational stores
• Event Hubs captures domain events for streaming data products

Documentation: - Microsoft Purview - Azure Synapse Analytics - Azure Data Lake Storage - Azure Data Share

GCP Implementation¶

Core Services¶

Project-Based Strategy¶

• Each domain gets its own GCP project
• Dataplex organizes data into lakes, zones, and assets across projects
• BigQuery datasets serve as data product output ports
• Analytics Hub enables cross-project data sharing without data movement
• Cloud IAM with organization-level policies for federated governance
• Pub/Sub topics for domain event publishing

Key Integration Points¶

• Dataplex auto-discovers and catalogs data across projects
• BigQuery authorized views and datasets for controlled sharing
• Analytics Hub listings for marketplace-style data product discovery
• Data Catalog tags for classification and business metadata

Documentation: - Google Cloud Dataplex - BigQuery - BigQuery Analytics Hub - Cloud Storage

Design Considerations¶

Domain Identification¶

• Align with business capabilities: Domains should map to business units or capabilities (sales, marketing, product, customer)
• Bounded contexts: Use domain-driven design (DDD) to identify natural boundaries
• Source-aligned domains: Own operational data (orders, transactions)
• Consumer-aligned domains: Create aggregated data products (analytics, reporting)
• Avoid too many domains: Start with 3-5 domains and expand as the organization matures

Data Product Design¶

• Define clear interfaces: Standardize how data products are consumed (SQL, API, files)
• Version schemas: Use semantic versioning for breaking changes
• Self-describing: Include metadata, documentation, and sample data
• Immutable: Append-only data with full history when possible
• SLA-driven: Define freshness, completeness, and availability guarantees

Platform Design¶

• Golden paths: Provide templates and tooling for common data product patterns
• Self-service provisioning: Domains should be able to create infrastructure without tickets
• Observability built-in: Monitoring, alerting, and quality checks as platform features
• Security by default: Encryption, access control, and audit logging as defaults

Data as a Product¶

Product Thinking for Data¶

Data Product Quality Dimensions¶

• Freshness: How recently was the data updated?
• Completeness: Are there missing values or records?
• Accuracy: Does the data reflect reality?
• Consistency: Is the data consistent across sources?
• Uniqueness: Are there duplicate records?
• Timeliness: Is data available when needed?

Data Product Metadata¶

Every data product should include: - Name and description: Clear, business-friendly naming - Owner and contact: Who to reach for questions - Schema: Field definitions, types, and constraints - SLA: Freshness, availability, and quality guarantees - Lineage: Where the data comes from and how it is transformed - Sample data: Examples for quick understanding - Usage examples: SQL queries, API calls, code snippets - Access request process: How to get access

Federated Governance¶

Governance Model¶

                    [Global Policies]
                    - Naming standards
                    - Security baseline
                    - Quality thresholds
                    - Interoperability
                          |
         +----------------+----------------+
         |                |                |
   [Domain: Sales]  [Domain: Product] [Domain: Customer]
   - Domain rules   - Domain rules    - Domain rules
   - Local quality  - Local quality   - Local quality
   - Access mgmt    - Access mgmt    - Access mgmt

Global vs Domain Policies¶

Automation¶

• Policy as Code: Define governance rules in version-controlled code
• Automated Quality Checks: Run data quality validations on every update
• Schema Validation: Enforce schema standards at ingestion time
• Access Auditing: Automatically log and review data access patterns
• Compliance Scanning: Detect PII and sensitive data automatically

Cost Estimation¶

AWS (Monthly Estimate per Domain)¶

Azure (Monthly Estimate per Domain)¶

GCP (Monthly Estimate per Domain)¶

Production Checklist¶

Organization Readiness¶

• Domain boundaries identified and agreed upon
• Domain data product owners assigned
• Data mesh principles communicated to stakeholders
• Organizational support and executive sponsorship secured
• Training plan for domain teams created

Platform Infrastructure¶

• Multi-account/project/subscription strategy implemented
• Self-serve provisioning templates available
• Data catalog deployed and accessible
• Data quality framework operational
• Monitoring and alerting configured
• CI/CD pipelines for data product deployment

Data Products¶

• Data product template and standards defined
• Schema registry or catalog configured
• At least one data product per domain published
• Documentation standards enforced
• SLAs defined and monitored
• Data product versioning strategy implemented

Governance¶

• Global governance policies documented
• Policy-as-code implemented and enforced
• Data classification and PII detection automated
• Access control model implemented (RBAC/ABAC)
• Compliance requirements mapped to controls
• Data lineage tracking operational

Operations¶

• Data quality dashboards available
• Incident response process for data issues defined
• Cross-domain data sharing tested and validated
• Performance benchmarks established
• Cost allocation per domain configured
• Regular data mesh maturity assessments scheduled

Related Guides: - Data Pipeline ETL Architecture - Lakehouse Architecture - Event-Driven Architecture