Databases¶

SQL, NoSQL, vector databases, data warehouses, lakehouses. The choice is rarely "which database" - it's "which database for which workload."

flowchart LR
  W{Workload}
  W -- transactional<br/>strong consistency --> SQL[Relational:<br/>RDS, Cloud SQL, Spanner]
  W -- key-value<br/>at scale --> KV[Wide-column:<br/>DynamoDB, Bigtable]
  W -- document<br/>flexible schema --> DOC[Document:<br/>Cosmos DB, Firestore, MongoDB]
  W -- semantic search<br/>RAG --> VEC[Vector:<br/>pgvector, Pinecone, Weaviate]
  W -- analytics<br/>columnar --> OLAP[(Warehouse:<br/>BigQuery, Redshift, Snowflake)]
  W -- streaming<br/>+ batch unified --> LH[Lakehouse:<br/>Databricks Delta / Iceberg]

Learn¶

• Eventual consistency - CAP, when EC is fine, when it bites
• Idempotency explained - retries, idempotency keys, common patterns
• Queues vs streams - SQS vs Kinesis (and equivalents), ordering, replay
• Embeddings and vector search - vector DB basics

Compare¶

• Databases (cloud-native) - RDS vs Cloud SQL vs SQL Database, DynamoDB vs Cosmos DB vs Firestore, BigQuery vs Redshift vs Synapse
• Vector databases - Pinecone, Weaviate, Qdrant, Milvus, pgvector, OpenSearch, Vertex Vector Search

Reference¶

• Architecture pattern: Data Pipeline / ETL
• Architecture pattern: Lakehouse
• Architecture pattern: Data Mesh
• Architecture pattern: CQRS / Event Sourcing
• Migration: database

Build¶

• Build a data pipeline
• Build a RAG pipeline - exercises pgvector

Certify¶

Cloud-side data certs - AWS Data Engineer Associate (DEA-C01) - AWS Database Specialty (DBS-C01) - retired, retained for credential holders - AWS Data Analytics Specialty (DAS-C01) - retired, retained - Azure Data Engineer (DP-203) - Azure Database Administrator (DP-300) - Azure Cosmos DB Developer (DP-420) - Azure Fabric Analytics Engineer (DP-600) - Azure Fabric Data Engineer (DP-700) - GCP Data Engineer - GCP Cloud Database Engineer

Vendor-specific - Databricks Data Engineer Associate - Databricks Data Engineer Professional - Snowflake SnowPro Core - Snowflake SnowPro Advanced Architect - Snowflake SnowPro Advanced Data Engineer - Snowflake SnowPro Advanced Administrator - MongoDB Associate Developer - MongoDB Associate DBA - MongoDB Associate Atlas Administrator - Confluent Certified Developer for Apache Kafka - Confluent Certified Administrator for Apache Kafka

Roadmap¶

Career paths live in the Data Engineer roadmap and Database Specialist roadmap.