Data engineering¶

Getting data from where it is produced to where it can be asked questions of, reliably and at a cost somebody approved. The databases topic covers where data rests; this one covers how it moves, what shape it arrives in, and how anyone knows it is right.

flowchart LR
  SRC[(Sources:<br/>app DBs, APIs, events, files)]
  SRC -->|batch: ETL / ELT| LAND
  SRC -->|streaming: Kafka, Kinesis, Pub/Sub| LAND
  LAND[Bronze<br/>raw, replayable] --> SIL[Silver<br/>cleaned, typed, joined]
  SIL --> GLD[Gold<br/>modelled, aggregated]
  GLD --> BI[BI and dashboards]
  GLD --> ML[ML features and training]
  QA{{Quality checks and lineage}} -.governs.- SIL
  QA -.governs.- GLD
  ORCH{{Orchestration: schedule, retry, backfill}} -.runs.- LAND
  ORCH -.runs.- SIL
  ORCH -.runs.- GLD

The shape nearly every data platform ends up with, whatever the vendor names. The dotted boxes are the two things that decide whether it is trusted: something that runs the steps in order, and something that checks the result.

Learn¶

Warehouses, lakes, and lakehouses - the three storage answers, and what a table format adds
ETL vs ELT - why the transform moved after the load, and when it should not
Batch vs streaming - event time, watermarks, and when streaming is not worth it
Data modeling for analytics - star schemas, grain, slowly changing dimensions
File formats and partitioning - Parquet, pruning, small files, and the bill
Data pipelines and orchestration - DAGs, retries, backfills, the silent zero
Data quality and lineage - the assertions that catch a wrong number before a dashboard does
SQL vs NoSQL - the operational stores most of this data is copied from
Queues vs streams - the transport underneath streaming pipelines
Eventual consistency - why a replica read can disagree with the source
Idempotency explained - the property that makes a retry or a backfill safe

Compare¶

Databases (cloud-native) - BigQuery vs Redshift vs Synapse, and the operational stores
Messaging and queues - Kafka, Kinesis, Pub/Sub, Event Hubs, SQS
Vector databases - the analytical store for embeddings
AI and ML services - where the feature pipelines end up

Reference¶

Architecture pattern: data pipeline / ETL
Architecture pattern: lakehouse
Architecture pattern: data mesh - decentralized ownership, domain data products
Architecture pattern: CQRS and event sourcing
Architecture pattern: event-driven architecture
Architecture pattern: AI/ML pipeline
Migration: database
Compliance guides - GDPR and the rest, where lineage stops being optional

Build¶

Build a data pipeline - ingestion through to a queryable table
Build a RAG pipeline - the embedding-and-retrieval variant
Set up a monitoring stack - the observability half of the argument

Certify¶

Cloud data engineering

AWS Data Engineer Associate (DEA-C01)
Azure Data Engineer (DP-203)
Azure Fabric Data Engineer (DP-700)
Azure Fabric Analytics Engineer (DP-600)
GCP Professional Data Engineer
AWS Data Analytics Specialty (DAS-C01) - retired, retained for credential holders

Platform-specific

Entry points and adjacent

Azure Data Fundamentals (DP-900) - the cheapest way to check the vocabulary has landed
Power BI Data Analyst (PL-300) - the consumption end
Azure Database Administrator (DP-300)
GCP Cloud Database Engineer
NVIDIA Accelerated Data Science (NCP-ADS)

Roadmap¶

The ordered path through these is the Data Engineer roadmap. For the storage-administration side, see the Database Specialist roadmap; for cost control over the warehouse bill, FinOps.

Related topics: Databases for choosing a store per workload, AI/ML systems for what consumes the feature tables, and Observability for the monitoring discipline the quality checks borrow from.