Design and Develop Data Processing¶

Overview¶

This domain covers ingesting and transforming data, developing batch and stream processing solutions, and managing pipelines. It represents 40-45% of the exam and is the largest domain, focusing on ETL/ELT implementation and data pipeline development.

Key Topics¶

Ingest and Transform Data¶

• Transform data with Apache Spark
• Transform data with Transact-SQL
• Transform data with Data Factory
• Transform data with Azure Synapse Pipelines
• Transform data with Stream Analytics
• Data cleansing operations
• Data splitting and shredding JSON
• Data encoding and decoding
• Error handling configuration for transformations
• Value normalization and denormalization
• Transform data with Scala
• Exploratory data analysis

Batch Processing Solutions¶

• Batch processing with Data Factory, Data Lake, Spark, Synapse Pipelines, PolyBase, Databricks
• Data pipeline creation
• Incremental data load design and implementation
• Slowly changing dimension design and development
• Security and compliance requirements handling
• Resource scaling
• Batch size configuration
• Test creation for data pipelines
• Jupyter/IPython notebook integration
• Data upsert operations
• Regression to previous state
• Exception handling design and configuration
• Batch retention configuration

Stream Processing Solutions¶

• Stream processing with Stream Analytics, Databricks, Event Hubs
• Spark structured streaming processing
• Performance and functional regression monitoring
• Windowed aggregate design and creation
• Schema drift handling
• Time series data processing
• Partition processing (across and within)
• Checkpoint and watermarking configuration
• Resource scaling for streaming
• Pipeline testing
• Analytical vs transactional optimization
• Interruption handling
• Exception handling
• Data upserts in streaming
• Archived stream data replay

Pipeline Management¶

• Batch triggering
• Failed batch load handling
• Batch load validation
• Data pipeline management in Data Factory/Synapse
• Data pipeline scheduling
• Version control for pipeline artifacts
• Spark job management in pipelines

Services Reference¶

Core Services¶

• Azure Data Factory
• Azure Synapse Analytics
• Azure Databricks
• Azure Stream Analytics
• Apache Spark
• Event Hubs
• PolyBase

Supporting Services¶

• Azure DevOps (for CI/CD)
• Git (for version control)
• Azure Monitor
• Log Analytics
• Power BI

Best Practices¶

Data Transformation¶

• Use Spark for large-scale transformations
• Use T-SQL for set-based operations in SQL pools
• Implement data quality checks at each stage
• Use schema validation before processing
• Handle null values consistently

Batch Processing¶

• Implement idempotent pipelines
• Use incremental loads to minimize processing
• Partition data for parallel processing
• Implement proper error handling and retry logic
• Monitor pipeline execution and performance
• Use metadata-driven pipelines for scalability

Stream Processing¶

• Design for exactly-once or at-least-once semantics
• Use windowing functions for time-based aggregations
• Implement checkpointing for fault tolerance
• Handle late-arriving data appropriately
• Monitor streaming lag and throughput

Pipeline Development¶

• Use parameters for configuration flexibility
• Implement logging and monitoring
• Version control all pipeline definitions
• Create reusable components and templates
• Implement CI/CD for pipeline deployment
• Test pipelines in non-production environments

Common Scenarios¶

Batch ETL Patterns¶

• Full load followed by incremental updates
• Change data capture (CDC) implementation
• SCD Type 2 updates in dimensions
• Fact table incremental loads
• Data quality validation and cleansing

Stream Processing Use Cases¶

• Real-time dashboard updates
• IoT telemetry processing
• Fraud detection systems
• Log analytics and monitoring
• Real-time recommendation engines

Data Integration¶

• Multi-source data consolidation
• Cloud-to-cloud data migration
• On-premises to cloud data replication
• API-based data ingestion
• File-based data processing

Pipeline Orchestration¶

• Complex workflow dependencies
• Parallel execution for performance
• Conditional execution based on data
• Scheduled and event-driven pipelines
• Cross-service orchestration

Study Tips¶

• Practice creating Data Factory pipelines with various activities
• Learn Spark DataFrame API and SQL in Python and Scala
• Hands-on experience with Stream Analytics query language
• Study windowing functions (tumbling, hopping, sliding, session)
• Practice implementing SCD Type 2 with Spark or T-SQL
• Understand Spark partitioning and shuffling concepts
• Learn Delta Lake operations (merge, time travel)
• Practice error handling patterns in pipelines
• Study incremental load patterns with watermarking
• Understand pipeline monitoring and alerting