Google Cloud Professional Data Engineer - Fact Sheet¶

Quick Reference¶

Exam Code: Professional Data Engineer Duration: 120 minutes (2 hours) Questions: 50-60 questions Passing Score: ~70% (not officially published) Cost: $200 USD Validity: 2 years Difficulty: ⭐⭐⭐⭐⭐ (Expert-level data engineering certification) Prerequisites: Recommended 3+ years of industry experience, including 1+ year designing and managing data solutions on GCP

Exam Domains¶

Core Data Engineering Principles¶

Data Engineering Framework¶

📖 Data Engineering on Google Cloud - Complete data engineering guide

Key Pillars: 1. Data Ingestion - Batch and streaming data collection 2. Data Processing - ETL/ELT transformations 3. Data Storage - Appropriate storage for use case 4. Data Analysis - Analytics and ML preparation 5. Data Governance - Security, compliance, quality

Essential Resources: - 📖 Data Lifecycle on Google Cloud - End-to-end data lifecycle - 📖 Big Data Reference Architectures - Common patterns - 📖 Data Analytics Best Practices - Analytics guidelines - 📖 Modern Data Architecture - Architecture patterns

Data Ingestion Services¶

Cloud Pub/Sub¶

Event Streaming and Messaging: - Global message queue for real-time and batch scenarios - At-least-once delivery guarantee - Message ordering within a single publisher - Pull and push subscriptions - Message retention: Default 7 days, up to 31 days - Dead letter topics for failed messages - 📖 Pub/Sub Overview - Architecture and concepts - 📖 Publisher and Subscriber - Publishing and consuming - 📖 Message Ordering - Guaranteed ordering - 📖 Schema Validation - Message validation - 📖 Dead Letter Topics - Error handling - 📖 Pub/Sub Lite - Cost-optimized regional messaging

Advanced Features: - Exactly-once delivery - Deduplication for subscriptions - Message filtering - Subscriber-side message filtering - Seek and replay - Replay messages from specific points - Snapshots - Capture subscription state for replay - 📖 Exactly-Once Delivery - Deduplication - 📖 Best Practices - Performance optimization

Storage Transfer Service¶

Data Transfer Solutions: - Transfer data from AWS S3, Azure Blob, HTTP/HTTPS sources - Scheduled and one-time transfers - Bandwidth throttling and filtering - 📖 Storage Transfer Service - Overview and setup - 📖 Transfer from AWS - AWS S3 migration - 📖 Transfer Appliance - Offline data transfer for petabyte-scale

BigQuery Data Transfer Service¶

Automated Data Imports: - Scheduled imports from SaaS applications (Google Ads, YouTube, etc.) - Partner integrations (Amazon S3, Teradata, Redshift) - Automated backfill and incremental updates - 📖 BigQuery Data Transfer Service - Complete guide - 📖 Supported Data Sources - Available connectors

Cloud Data Fusion¶

Visual ETL Pipeline Builder: - No-code/low-code data integration - Pre-built connectors for 150+ sources - Visual pipeline designer with drag-and-drop - CDAP (Cask Data Application Platform) based - 📖 Cloud Data Fusion - Overview and architecture - 📖 Pipeline Creation - Building pipelines - 📖 Wrangler - Interactive data preparation

Data Processing Services¶

Dataflow¶

Unified Stream and Batch Processing: - Apache Beam-based serverless processing - Auto-scaling from zero to thousands of workers - Exactly-once processing semantics - Late data handling with windowing and triggers - Flexible Resource Scheduling (FlexRS) for batch cost savings - 📖 Dataflow Overview - Architecture fundamentals - 📖 Apache Beam Programming Guide - Beam concepts - 📖 Dataflow Templates - Pre-built templates - 📖 Streaming Engine - Streaming optimization - 📖 Windowing - Time-based grouping - 📖 Triggers and Watermarks - Late data handling

Advanced Patterns: - Side inputs - Broadcast data to all workers - State and Timers - Stateful processing - Splittable DoFn - Efficiently process large elements - 📖 Dataflow Best Practices - Performance optimization - 📖 Dataflow SQL - SQL-based pipelines - 📖 FlexRS - Cost-optimized batch processing

Dataproc¶

Managed Hadoop and Spark: - Fast cluster provisioning (90 seconds) - Autoscaling clusters based on workload - Preemptible VMs for worker nodes (80% cost savings) - Component gateway for web UIs - Job scheduling and workflow management - 📖 Dataproc Overview - Architecture and use cases - 📖 Cluster Creation - Cluster configuration - 📖 Autoscaling - Dynamic scaling - 📖 Workflow Templates - Job orchestration - 📖 Initialization Actions - Custom cluster setup

Ecosystem Support: - Apache Spark, Hadoop, Pig, Hive, Presto - Jupyter and Zeppelin notebooks - Serverless Spark - Fully managed Spark without clusters - 📖 Dataproc Jobs - Job submission - 📖 Dataproc Hub - Notebook-based workflows - 📖 Serverless Spark - Serverless batch processing - 📖 Metastore - Managed Hive metastore

Optimization Strategies: - Preemptible workers for cost optimization - Enhanced flexibility mode for reliability - Local SSDs for shuffle operations - 📖 Best Practices - Performance guidelines

Cloud Composer¶

Managed Apache Airflow: - Workflow orchestration for data pipelines - DAG (Directed Acyclic Graph) based workflows - Rich operator ecosystem (GCP, AWS, Azure) - Built-in monitoring and logging - Environment versioning and upgrades - 📖 Cloud Composer Overview - Architecture fundamentals - 📖 Creating DAGs - Workflow creation - 📖 GCP Operators - GCP integrations - 📖 Composer Environments - Version management - 📖 Best Practices - Performance optimization

Common Patterns: - Data pipeline orchestration (ETL/ELT) - Cross-cloud workflow coordination - ML pipeline automation - Data validation and quality checks - 📖 Triggering DAGs - Event-driven workflows

Dataprep by Trifacta¶

Visual Data Preparation: - Interactive UI for data exploration - Smart suggestions for data transformations - No-code data wrangling - Automatic data profiling and statistics - 📖 Dataprep Documentation - Complete guide - 📖 Recipe Creation - Building transformations

Data Storage Services¶

BigQuery¶

Serverless Data Warehouse: - Petabyte-scale SQL analytics - Separation of compute and storage - Standard SQL (ANSI 2011) support - Automatic backup and disaster recovery - Integration with BI tools (Looker, Tableau, Power BI) - 📖 BigQuery Overview - Architecture fundamentals - 📖 BigQuery Storage - Columnar storage - 📖 Query Execution - Query processing - 📖 Slots and Reservations - Capacity management - 📖 Pricing Models - On-demand vs flat-rate

Table Design and Optimization: - Partitioning - Time-based, integer range, or ingestion time - Clustering - Sort data within partitions (up to 4 columns) - Nested and Repeated Fields - STRUCT and ARRAY types - Search Indexes - Full-text search capabilities - 📖 Partitioned Tables - Partition strategies - 📖 Clustered Tables - Clustering design - 📖 Nested and Repeated Data - Complex data types - 📖 Search Indexes - Text search optimization

Advanced Features: - Materialized Views - Pre-computed query results - BI Engine - In-memory analytics acceleration - External Tables - Query data in Cloud Storage, Bigtable, Sheets - Federated Queries - Query across Cloud SQL, Spanner - Wildcard Tables - Query multiple tables with patterns - 📖 Materialized Views - View optimization - 📖 BI Engine - Query acceleration - 📖 External Tables - External data sources - 📖 Federated Queries - Cross-service queries

Data Loading Patterns: - Batch loading (CSV, JSON, Avro, Parquet, ORC) - Streaming inserts via API - Storage Write API for high-throughput streaming - BigQuery Data Transfer Service - 📖 Loading Data - Ingestion methods - 📖 Streaming Inserts - Real-time loading - 📖 Storage Write API - High-throughput streaming

Cost Optimization: - Query cost: $5 per TB processed (on-demand) - Use partitioning and clustering to reduce scanned data - Preview queries to estimate cost - Set maximum bytes billed limits - Use flat-rate pricing for predictable costs - 📖 Cost Optimization - Cost reduction strategies - 📖 Query Optimization - Performance tuning

Cloud Bigtable¶

Wide-Column NoSQL Database: - Petabyte-scale, sub-10ms latency - Ideal for time-series, IoT, financial, AdTech - HBase API compatible (easy migration) - Linear scalability with node count - Multi-cluster replication for HA and geo-distribution - 📖 Bigtable Overview - Architecture and use cases - 📖 Storage Model - Data organization - 📖 Instances and Clusters - Infrastructure setup

Schema Design Best Practices: - Row key design is critical (determines performance) - Avoid monotonically increasing keys (hotspots) - Reverse timestamps for time-series data - Field promotion for frequently accessed data - Keep column families small - 📖 Schema Design - Design patterns - 📖 Schema Design for Time-Series - Time-series patterns - 📖 Row Key Design - Performance optimization

Performance and Scaling: - Autoscaling based on CPU and storage utilization - Replication for read scaling and disaster recovery - App profiles for traffic routing - 📖 Performance Tuning - Optimization guide - 📖 Replication - Multi-cluster setup - 📖 App Profiles - Traffic management

Integration with Data Processing: - Native Dataflow connectors - Hadoop/Spark integration via HBase API - 📖 Dataflow Integration - Streaming pipelines - 📖 Beam Connector - Apache Beam integration

Cloud Storage¶

Object Storage for Data Lakes: - Unlimited scale, 11 9's durability - Storage classes: Standard, Nearline, Coldline, Archive - Lifecycle management for automatic tiering - Versioning and retention policies - Strong consistency for all operations - 📖 Cloud Storage Overview - Complete guide - 📖 Storage Classes - Class comparison - 📖 Lifecycle Management - Automated policies - 📖 Object Versioning - Version control

Data Lake Patterns: - Raw zone → Processed zone → Curated zone - Partition data by date in folder structure - Use Parquet or Avro for analytics workloads - Compress data to reduce storage and transfer costs - 📖 Data Lake Best Practices - Architecture patterns - 📖 Object Notification - Event-driven processing

Performance Optimization: - Parallel composite uploads for large files - Regional vs multi-regional for latency/availability trade-offs - Request rate limits: 5000 writes per second per prefix - 📖 Request Rate and Access Distribution - Performance guidelines - 📖 Turbo Replication - Fast geo-replication

Cloud SQL¶

Managed Relational Databases: - MySQL, PostgreSQL, SQL Server support - High availability with automatic failover - Read replicas for scaling reads - Point-in-time recovery - Integration with Dataflow, Dataproc, BigQuery - 📖 Cloud SQL Overview - Architecture guide - 📖 High Availability - HA configuration - 📖 Replication - Read replicas and cross-region - 📖 Backup and Recovery - Backup strategies

Data Engineering Use Cases: - Operational data store for applications - ETL source and target - Metadata storage for pipelines - Federation target from BigQuery - 📖 Federated Queries from BigQuery - Cross-service queries

Cloud Spanner¶

Globally Distributed SQL Database: - Horizontal scalability with ACID transactions - 99.999% availability SLA (multi-region) - Automatic sharding and replication - Strong consistency globally - SQL support with extensions - 📖 Cloud Spanner Overview - Architecture concepts - 📖 Schema Design - Best practices - 📖 Replication - Multi-region setup - 📖 Query Best Practices - Performance optimization

Data Engineering Integration: - Dataflow connector for batch and streaming - Federated queries from BigQuery - Change streams for CDC pipelines - 📖 Dataflow Integration - Pipeline integration - 📖 Change Streams - Change data capture

Firestore¶

NoSQL Document Database: - Serverless with automatic scaling - Real-time synchronization - Offline support for mobile/web - ACID transactions - Native and Datastore modes - 📖 Firestore Overview - Complete guide - 📖 Data Model - Documents and collections - 📖 Queries - Query capabilities - 📖 Dataflow Integration - Streaming pipelines

Memorystore¶

Managed In-Memory Databases: - Memorystore for Redis: Caching, session store, pub/sub - Memorystore for Memcached: High-performance caching - Sub-millisecond latency - High availability with automatic failover - 📖 Memorystore for Redis - Redis overview - 📖 Memorystore for Memcached - Memcached overview

Machine Learning and AI Services¶

Vertex AI¶

Unified ML Platform: - AutoML for no-code model training - Custom training with pre-built containers - Model deployment and serving (online/batch) - Feature Store for feature management - ML Pipelines for workflow automation - 📖 Vertex AI Overview - Platform overview - 📖 AutoML - Automated ML - 📖 Custom Training - Custom models - 📖 Feature Store - Feature management - 📖 ML Pipelines - Workflow orchestration - 📖 Model Monitoring - Drift detection

Vertex AI Workbench: - Jupyter notebook environment - Pre-configured for ML frameworks - Integration with Git and Docker - 📖 Vertex AI Workbench - Notebook environment

BigQuery ML¶

SQL-Based Machine Learning: - Train models using SQL (no Python required) - Supported models: Linear/logistic regression, K-means, time-series, AutoML - Model export to Vertex AI for deployment - Integration with TensorFlow and XGBoost - 📖 BigQuery ML Overview - BQML fundamentals - 📖 Model Types - Available algorithms - 📖 CREATE MODEL - Model training syntax - 📖 Model Evaluation - Model assessment - 📖 Hyperparameter Tuning - Optimization

Common Use Cases: - Customer churn prediction - Product recommendations - Demand forecasting - Anomaly detection - 📖 BQML Tutorials - Hands-on examples

Pre-trained AI APIs¶

Ready-to-Use AI Services: - Vision API: Image analysis and OCR - Natural Language API: Text analysis and entity extraction - Translation API: Language translation - Speech-to-Text and Text-to-Speech - Video Intelligence API: Video analysis - 📖 Vision API - Image analysis - 📖 Natural Language API - Text understanding - 📖 Translation API - Language translation - 📖 Speech-to-Text - Audio transcription - 📖 Video Intelligence - Video analysis

TensorFlow and AI Platform¶

Deep Learning Infrastructure: - TensorFlow Enterprise with Long-Term Support - Distributed training with GPUs/TPUs - TensorFlow Extended (TFX) for production ML pipelines - 📖 TensorFlow Enterprise - Enterprise TF - 📖 Cloud TPU - Tensor Processing Units

Data Governance and Security¶

Data Catalog¶

Metadata Management: - Automatic metadata discovery for BigQuery, Pub/Sub - Custom metadata with tags and templates - Data lineage tracking - Search and discovery across data assets - 📖 Data Catalog Overview - Metadata management - 📖 Tag Templates - Custom metadata - 📖 Data Lineage - Lineage tracking

Data Loss Prevention (DLP)¶

Sensitive Data Discovery and Protection: - Automatic PII detection (credit cards, SSN, emails) - Custom info types for organization-specific data - De-identification techniques (masking, tokenization, encryption) - Inspection templates and job triggers - 📖 DLP Overview - DLP fundamentals - 📖 Inspecting Data - Data inspection - 📖 De-identification - Data protection - 📖 InfoTypes - Detectable data types

Identity and Access Management (IAM)¶

Data Access Control: - Resource-level permissions (project, dataset, table) - Column-level security in BigQuery - Row-level security with authorized views - VPC Service Controls for data perimeter security - 📖 IAM Overview - IAM fundamentals - 📖 BigQuery IAM - Dataset permissions - 📖 Column-Level Security - Fine-grained access - 📖 Row-Level Security - Row filtering - 📖 VPC Service Controls - Data exfiltration protection

Encryption and Key Management¶

Data Protection: - Default encryption at rest (Google-managed keys) - Customer-Managed Encryption Keys (CMEK) with Cloud KMS - Customer-Supplied Encryption Keys (CSEK) - Encryption in transit (TLS) - 📖 Encryption at Rest - Default protection - 📖 Cloud KMS - Key management - 📖 CMEK - Customer-managed keys - 📖 BigQuery Encryption - BQ encryption options

Data Quality and Validation¶

Ensuring Data Integrity: - Great Expectations integration with Dataflow - Dataprep data quality rules - BigQuery data quality checks with SQL - Cloud Composer for validation pipelines - 📖 Data Validation Best Practices - Validation patterns

Monitoring and Optimization¶

Cloud Monitoring¶

Infrastructure and Pipeline Monitoring: - Pre-built dashboards for GCP services - Custom metrics from applications - Alerting policies with notification channels - Uptime checks for service availability - 📖 Cloud Monitoring - Complete guide - 📖 Dataflow Monitoring - Pipeline metrics - 📖 BigQuery Monitoring - Query monitoring - 📖 Pub/Sub Monitoring - Message queue metrics

Cloud Logging¶

Centralized Log Management: - Automatic collection of GCP service logs - Log sinks to BigQuery for analysis - Log-based metrics for custom monitoring - Audit logs for compliance - 📖 Cloud Logging - Logging overview - 📖 Log Sinks - Export to BigQuery/Storage - 📖 Audit Logs - Compliance logging - 📖 Log Analytics - SQL-based log analysis

Performance Optimization¶

BigQuery Optimization: - Avoid SELECT * - specify columns - Filter early with WHERE clauses - Use partitioning and clustering - Denormalize data when appropriate - Use approximate aggregation functions - 📖 Query Best Practices - Query optimization - 📖 SQL Anti-Patterns - What to avoid

Dataflow Optimization: - Use windowing to reduce state size - Tune worker resources (CPU, memory, disk) - Enable Streaming Engine for streaming jobs - Use side inputs for broadcast joins - Batch elements with GroupIntoBatches - 📖 Dataflow Best Practices - Pipeline optimization - 📖 Pipeline Performance - Troubleshooting guide

Bigtable Optimization: - Design row keys to avoid hotspots - Pre-split tables for write-heavy workloads - Use caching for frequently accessed data - Monitor key metrics: CPU, storage, latency - 📖 Bigtable Performance Guide - Optimization strategies

Data Migration Strategies¶

Migration Patterns¶

The 5 Rs of Migration: 1. Rehost - Lift-and-shift (VM migration) 2. Replatform - Minor optimizations (Dataproc instead of on-prem Hadoop) 3. Refactor - Re-architect for cloud-native (Dataflow, BigQuery) 4. Retire - Decommission unnecessary systems 5. Retain - Keep on-premises temporarily

Migration Tools: - 📖 Data Migration Guide - Hadoop migration - 📖 BigQuery Migration Service - SQL translation - 📖 Database Migration Service - Database replication

Hadoop to GCP Migration¶

Migration Path: - HDFS → Cloud Storage - MapReduce/Spark → Dataproc or Dataflow - Hive/Impala → BigQuery - HBase → Bigtable - 📖 Hadoop to GCP - Migration guide - 📖 Dataproc Best Practices - Job migration

Data Warehouse Migration¶

Migration to BigQuery: - Schema translation tools - Data extraction and loading strategies - Query translation (Teradata, Redshift, SQL Server) - Incremental migration approach - 📖 Data Warehouse Migration - Complete guide - 📖 BigQuery Migration Service - Automated translation

Common Data Architecture Patterns¶

Pattern 1: Real-Time Analytics Pipeline¶

Architecture: - Ingestion - Pub/Sub for event streaming - Processing - Dataflow for stream processing and aggregation - Storage - BigQuery for analytics, Bigtable for operational queries - Visualization - Looker or Data Studio for dashboards

Key Services: - Pub/Sub with exactly-once delivery - Dataflow with Streaming Engine - BigQuery streaming inserts or Storage Write API - 📖 Streaming Analytics Reference Architecture - Complete pattern

Pattern 2: Batch ETL Pipeline¶

Architecture: - Source - Cloud Storage (CSV, JSON, Avro, Parquet) - Orchestration - Cloud Composer (Airflow) DAGs - Processing - Dataflow or Dataproc for transformations - Destination - BigQuery for analytics

Key Services: - Cloud Composer for scheduling - Dataflow for serverless ETL - BigQuery for data warehouse - 📖 Batch ETL Pattern - ETL architecture

Pattern 3: Data Lake Architecture¶

Architecture: - Raw Zone - Cloud Storage (original formats) - Processed Zone - Cloud Storage (cleaned, validated) - Curated Zone - Cloud Storage or BigQuery (analytics-ready) - Metadata - Data Catalog for discovery

Key Services: - Cloud Storage for all zones - Dataflow for processing between zones - BigQuery external tables for ad-hoc queries - 📖 Data Lake on GCP - Complete architecture

Pattern 4: Machine Learning Pipeline¶

Architecture: - Data Prep - Dataflow for feature engineering - Training - Vertex AI for model training - Serving - Vertex AI Endpoints for online prediction - Orchestration - Vertex AI Pipelines or Cloud Composer

Key Services: - BigQuery for feature store - Vertex AI for ML lifecycle - Cloud Storage for model artifacts - 📖 MLOps Architecture - ML pipelines

Pattern 5: Hybrid and Multi-Cloud Data¶

Architecture: - Connectivity - Cloud VPN or Interconnect - Data Transfer - Storage Transfer Service - Processing - Dataproc or Dataflow - Multi-Cloud - Pub/Sub for message routing

Key Services: - Cloud Interconnect for dedicated connectivity - Transfer Service for cross-cloud data movement - BigQuery Omni for multi-cloud analytics - 📖 Hybrid Data Processing - Hybrid patterns

Exam Scenarios and Solutions¶

Scenario 1: High-Volume Streaming Data¶

Requirements: Ingest millions of IoT events per second, real-time dashboards

Solution: - Pub/Sub for ingestion (regional Pub/Sub Lite for cost optimization) - Dataflow for aggregation and windowing - BigQuery for historical analysis - Bigtable for low-latency operational queries

Key Decision: Use Pub/Sub Lite if regional availability is acceptable for cost savings

Scenario 2: Batch Analytics Pipeline¶

Requirements: Daily ETL from multiple sources, SQL transformations, reporting

Solution: - Cloud Composer for orchestration - Dataflow or BigQuery SQL for transformations - BigQuery for data warehouse - Materialized views for dashboard performance

Key Decision: Use BigQuery SQL for transformations if logic is simple; Dataflow for complex multi-source joins

Scenario 3: Time-Series Data at Scale¶

Requirements: Store billions of time-series records, low-latency reads

Solution: - Bigtable with reverse timestamp row keys - Dataflow for real-time ingestion - Precomputed aggregations in BigQuery for analytics - App profiles for workload isolation

Key Decision: Row key design critical - reverse timestamp to avoid hotspots

Scenario 4: Cost-Optimized Data Lake¶

Requirements: Store petabytes of data, infrequent access, cost-sensitive

Solution: - Cloud Storage Nearline/Coldline/Archive - Lifecycle policies for automatic tiering - BigQuery external tables for ad-hoc queries - Federated queries to avoid data movement

Key Decision: Use Archive class for data accessed < 1/year; external tables to avoid storage duplication

Scenario 5: Data Governance and Compliance¶

Requirements: PII detection, access control, audit trail, data lineage

Solution: - DLP for automatic PII detection and de-identification - BigQuery column-level and row-level security - Data Catalog for metadata and lineage - VPC Service Controls for data perimeter

Key Decision: Combine DLP with BigQuery authorized views for dynamic data masking

Scenario 6: Machine Learning on Large Datasets¶

Requirements: Train models on TB-scale data, feature management, serving

Solution: - BigQuery for feature engineering with SQL - BigQuery ML for simple models, Vertex AI for complex - Feature Store for feature serving - Vertex AI Endpoints for online prediction

Key Decision: Use BigQuery ML if SQL-based features work; Vertex AI for custom TensorFlow/PyTorch

Exam Tips and Strategy¶

Keywords to Watch¶

Question Patterns: - "Real-time" or "streaming" → Pub/Sub + Dataflow + BigQuery/Bigtable - "Batch" or "scheduled" → Cloud Composer + Dataflow/Dataproc + BigQuery - "Cost-effective" → Preemptible VMs, Dataproc autoscaling, BigQuery partitioning, FlexRS - "Low latency" → Bigtable, Memorystore, BigQuery BI Engine - "Petabyte-scale analytics" → BigQuery - "Time-series data" → Bigtable with proper row key design - "SQL analytics" → BigQuery - "Complex transformations" → Dataflow with Apache Beam - "Simple transformations" → BigQuery SQL, Dataprep - "Machine learning" → BigQuery ML (SQL), Vertex AI (custom) - "Data discovery" → Data Catalog - "PII protection" → Cloud DLP - "Exactly-once processing" → Pub/Sub + Dataflow

Service Selection Decision Trees¶

Ingestion Decision:

Streaming data?
├─ YES → Pub/Sub → Dataflow → BigQuery/Bigtable
│  └─ Regional and cost-sensitive? → Pub/Sub Lite
└─ NO → Batch ingestion
   ├─ From SaaS apps → BigQuery Data Transfer Service
   ├─ From Cloud Storage → BigQuery load job
   ├─ From other clouds → Storage Transfer Service
   └─ Scheduled ETL → Cloud Composer

Processing Decision:

Processing type?
├─ Streaming → Dataflow (Apache Beam)
├─ Batch ETL
│  ├─ Simple SQL → BigQuery SQL
│  ├─ Visual/no-code → Cloud Data Fusion or Dataprep
│  └─ Complex → Dataflow or Dataproc
└─ Hadoop/Spark existing code → Dataproc

Storage Decision:

Data type?
├─ Analytics/SQL → BigQuery
├─ Time-series/IoT
│  ├─ Low latency → Bigtable
│  └─ Analytics → BigQuery with partitioning
├─ Relational transactional
│  ├─ Global scale → Cloud Spanner
│  └─ Regional → Cloud SQL
├─ Document/NoSQL → Firestore
├─ Object/file → Cloud Storage
└─ Caching → Memorystore

Time Management¶

• 120 minutes ÷ 50 questions = 2.4 minutes per question
• First pass: Answer confident questions (60 minutes)
• Second pass: Tackle scenario-based questions (45 minutes)
• Final pass: Review flagged questions (15 minutes)

Common Traps¶

• ❌ Choosing complex solutions when simple ones work (BigQuery SQL vs Dataflow)
• ❌ Not considering cost constraints (on-demand vs flat-rate, preemptible workers)
• ❌ Ignoring latency requirements (Bigtable vs BigQuery)
• ❌ Poor Bigtable row key design (monotonically increasing = hotspot)
• ❌ Not using partitioning/clustering in BigQuery
• ❌ Forgetting about managed alternatives (Dataflow vs self-managed Spark)
• ❌ Not recognizing when to use external tables vs loading data
• ❌ Mixing up Pub/Sub delivery guarantees

Study Checklist¶

Knowledge Areas: - [ ] Understand when to use BigQuery vs Bigtable vs Cloud SQL - [ ] Know Pub/Sub message delivery guarantees and patterns - [ ] Can design Dataflow pipelines with windowing and triggers - [ ] Understand BigQuery partitioning and clustering strategies - [ ] Know Bigtable row key design patterns - [ ] Familiar with Cloud Composer DAG creation - [ ] Understand BigQuery ML capabilities and limitations - [ ] Know data migration strategies (5 Rs) - [ ] Understand DLP and data governance tools - [ ] Can optimize costs for data processing workloads

Hands-On Skills: - [ ] Create streaming pipelines with Pub/Sub and Dataflow - [ ] Design and optimize BigQuery tables - [ ] Build batch ETL with Cloud Composer - [ ] Implement Bigtable schema for time-series data - [ ] Use BigQuery ML to train and deploy models - [ ] Configure IAM for data access control - [ ] Set up data quality validation pipelines - [ ] Monitor and troubleshoot Dataflow jobs

Preparation: - [ ] Build end-to-end data pipelines on GCP - [ ] Complete official practice exam (80%+ score target) - [ ] Review case studies in exam guide - [ ] Read all linked documentation sections - [ ] Practice with gcloud, bq, and gsutil CLI tools - [ ] Understand cost implications of design decisions

Pro Tip: The Professional Data Engineer exam heavily tests your ability to choose the right service for the use case. Always consider: data volume, latency requirements, cost constraints, processing complexity, and operational overhead. Practice building real pipelines to internalize when each service shines!

Documentation Count: This fact sheet contains 100+ embedded documentation links to official Google Cloud documentation.

Good luck! This certification demonstrates expert-level data engineering skills on Google Cloud Platform.