Analytics Workloads on Azure¶

Overview¶

Analytics workloads process and analyze large volumes of data to extract insights for business decisions. Azure provides comprehensive services for data ingestion, processing, storage, and visualization across batch and real-time analytics scenarios.

Data Ingestion and Processing¶

Batch vs Real-Time Ingestion¶

Batch Ingestion¶

• Definition: Collecting and processing data in scheduled intervals
• Characteristics:
• Processes large volumes at once
• Scheduled or trigger-based execution
• Higher latency (minutes to hours)
• Cost-effective for large datasets

• Optimized for throughput over latency

• Common Patterns:

• Daily/hourly data imports
• End-of-day processing
• Periodic data synchronization

• Historical data analysis

• Use Cases:

• Financial reporting and reconciliation
• Data warehouse ETL processes
• Monthly/quarterly analytics
• Backup and archival processes
• Machine learning model training

Real-Time Ingestion¶

• Definition: Processing data immediately as it arrives
• Characteristics:
• Continuous data stream processing
• Low latency (milliseconds to seconds)
• Immediate insights and actions
• Higher infrastructure complexity

• Designed for streaming data

• Common Patterns:

• Event streaming
• Change data capture (CDC)
• Message queuing

• Continuous data feeds

• Use Cases:

• Fraud detection
• Real-time dashboards
• IoT sensor monitoring
• Social media sentiment analysis
• Stock trading platforms
• Gaming leaderboards

Azure Data Factory¶

Overview¶

• Cloud-based data integration service
• Orchestrate and automate data movement and transformation
• Code-free visual ETL/ELT tool
• Serverless data integration

Key Components¶

Pipelines: - Logical grouping of activities - Workflow that performs a task - Can be scheduled or triggered - Support for control flow (if-then, loops, variables)

Activities: - Processing steps in pipeline - Types: - Data Movement: Copy data between sources - Data Transformation: Transform data (Databricks, HDInsight, SQL) - Control Flow: If condition, ForEach, Until, Wait

Datasets: - Named view of data - Point to data in data stores - Define structure and location - Used as input/output for activities

Linked Services: - Connection strings to data sources - Define connectivity information - Similar to connection strings - Support for many data sources

Triggers: - Schedule trigger: Time-based execution - Tumbling window trigger: Periodic intervals - Event-based trigger: Respond to events (blob creation)

Common Scenarios¶

• Data Migration: Move data to Azure
• ETL/ELT: Transform data for analytics
• Data Integration: Combine data from multiple sources
• Incremental Loading: Process only changed data
• Orchestration: Coordinate complex workflows

Integration Runtime¶

• Azure IR: Cloud data movement and transformation
• Self-hosted IR: On-premises and cloud data movement
• Azure-SSIS IR: Lift-and-shift SSIS packages

Use Cases¶

• Data warehouse loading
• Data lake ingestion
• Migrating on-premises data to cloud
• Orchestrating complex data workflows
• Scheduling and monitoring data pipelines

Azure Stream Analytics¶

Overview¶

• Real-time analytics service
• Process streaming data from multiple sources
• SQL-like query language
• Serverless with automatic scaling
• Low-code stream processing

Input Sources¶

• Azure Event Hubs: High-throughput event ingestion
• Azure IoT Hub: IoT device telemetry
• Azure Blob Storage: Reference data
• Azure Data Lake Storage: Static or slowly changing data

Output Targets¶

• Azure SQL Database
• Azure Synapse Analytics
• Blob Storage and Data Lake Storage
• Event Hubs (for chaining)
• Power BI (for real-time dashboards)
• Cosmos DB
• Azure Functions

Query Language¶

• SQL-like syntax for stream processing
• Temporal operations (windowing functions)
• Aggregations over time windows
• JOIN with reference data
• Pattern detection

Windowing Functions¶

Tumbling Window: - Fixed-size, non-overlapping time segments - Every event belongs to exactly one window - Example: 5-minute windows for aggregation

Hopping Window: - Fixed-size, overlapping windows - Events can belong to multiple windows - Example: 10-minute window, hopping every 5 minutes

Sliding Window: - Windows created when events occur - Only outputs when event enters or exits - Example: Events within last 10 minutes

Session Window: - Variable-size windows based on activity - Gaps of inactivity define window boundaries - Example: User session analysis

Use Cases¶

• Real-time dashboards and monitoring
• IoT device telemetry processing
• Fraud detection alerts
• Real-time analytics on streaming data
• Anomaly detection
• Log analysis and monitoring

Azure Event Hubs¶

Overview¶

• Big data streaming platform
• Event ingestion service
• Handles millions of events per second
• Fully managed PaaS
• Distributed streaming platform

Key Concepts¶

Events: - Small packets of information - Contains data payload and metadata - Immutable once sent

Event Publishers: - Send events to Event Hubs - HTTPS, AMQP, or Kafka protocol - Can be IoT devices, applications, services

Partitions: - Organize events into sequences - Scale unit for parallelization - Events with same partition key go to same partition - Typically 2-32 partitions

Consumer Groups: - View of entire event hub - Allows multiple consuming applications - Each consumer reads independently - Default consumer group always exists

Event Processors: - Consume and process events - Checkpoint progress - Handle load balancing across consumers

Features¶

• Capture: Automatically save events to Blob/Data Lake
• Auto-inflate: Automatically scale throughput units
• Geo-disaster recovery: Metadata replication
• Kafka compatibility: Use Kafka APIs with Event Hubs
• Schema Registry: Centralized schema management

Tiers¶

• Basic: 1 consumer group, 100 connections
• Standard: 20 consumer groups, 1000 connections, capture
• Premium: Dedicated resources, 100 consumer groups
• Dedicated: Single-tenant deployment

Use Cases¶

• Telemetry and distributed data streaming
• Event sourcing architectures
• IoT device data ingestion
• Log aggregation
• Real-time data pipelines
• Live dashboarding

Azure IoT Hub¶

Overview¶

• Managed service for IoT scenarios
• Bi-directional communication with IoT devices
• Built-in device management
• Secure device connectivity
• Per-device authentication

Key Features¶

Device Management: - Device registration and identity - Device twins (state and configuration) - Direct methods for device control - Firmware and configuration updates

Communication: - Device-to-cloud messaging - Cloud-to-device messaging - File uploads from devices - Multiple protocols: MQTT, AMQP, HTTPS

Security: - Per-device security credentials - X.509 certificates or SAS tokens - TLS/SSL encryption - Azure AD integration

Routing: - Route messages to different endpoints - Filter messages based on properties - Send to Event Hubs, Service Bus, Storage, Event Grid

Integration: - Azure Stream Analytics for real-time processing - Azure Functions for serverless processing - Azure Machine Learning for predictions - Power BI for visualization

Tiers¶

• Free: Limited messaging for development
• Basic: Device-to-cloud messaging, no cloud-to-device
• Standard: All features including cloud-to-device

Use Cases¶

• IoT device connectivity and management
• Predictive maintenance
• Remote monitoring and control
• Asset tracking
• Smart buildings and cities
• Industrial IoT

Comparison: Event Hubs vs IoT Hub¶

Analytical Data Stores¶

Data Warehouses vs Data Lakes¶

Data Warehouses¶

• Structure: Structured, processed data
• Schema: Schema-on-write (defined before loading)
• Data Types: Primarily structured data
• Users: Business analysts, BI professionals
• Processing: Optimized for SQL queries and reporting
• Agility: Less agile, schema changes are complex
• Storage: More expensive, optimized storage
• Use Cases: Business reporting, BI dashboards, historical analysis

Data Lakes¶

• Structure: Raw, unprocessed data in native format
• Schema: Schema-on-read (defined when reading)
• Data Types: All types (structured, semi-structured, unstructured)
• Users: Data scientists, data engineers, developers
• Processing: Flexible processing with various tools
• Agility: High agility, store first, process later
• Storage: Lower cost, commodity storage
• Use Cases: Big data analytics, machine learning, data exploration

Modern Approach: Data Lakehouse¶

• Combines best of both worlds
• Structured data on data lake storage
• ACID transactions on data lake
• Azure Synapse Analytics provides this capability

Azure Synapse Analytics¶

Overview¶

• Unified analytics platform
• Combines data warehousing and big data analytics
• Formerly SQL Data Warehouse
• Integrated with Power BI, Azure ML, and other services

Key Components¶

Dedicated SQL Pools: - Massively parallel processing (MPP) data warehouse - Provisioned compute resources - Optimized for complex queries on large datasets - Columnar storage with compression

Serverless SQL Pools: - On-demand SQL queries - Pay-per-query pricing - Query data in data lake without loading - No infrastructure management

Apache Spark Pools: - Big data processing with Spark - Scala, Python, .NET, R support - Integrated notebooks - Machine learning capabilities

Pipelines: - Built-in data integration (similar to Data Factory) - Orchestrate and schedule workflows - Code-free visual development

Synapse Studio: - Unified workspace - Single interface for all tasks - Collaborative development - Integrated monitoring and management

Data Distribution Strategies¶

Round Robin: - Evenly distributes rows across distributions - Fast loading - Not optimized for queries - Use for staging tables

Hash Distribution: - Distributes based on hash of column value - Best for large fact tables - Optimizes JOIN and GROUP BY performance - Choose distribution column carefully

Replicated: - Full copy on each compute node - Best for small dimension tables - Eliminates data movement - Faster joins with fact tables

Use Cases¶

• Enterprise data warehousing
• Big data analytics
• Real-time and batch analytics
• Data exploration and discovery
• Machine learning at scale
• Integrated analytics solutions

Azure Data Lake Storage Gen2¶

Overview¶

• Built on Azure Blob Storage
• Optimized for big data analytics
• Hierarchical namespace for directory structure
• Hadoop-compatible
• Enterprise-grade security

Key Features¶

Hierarchical Namespace: - Directory and file-level operations - Better performance for analytics workloads - Atomic directory operations - Organized data management

Performance: - Optimized for parallel processing - High throughput for big data workloads - Efficient metadata operations

Security: - POSIX-compliant ACLs - Role-based access control (RBAC) - Encryption at rest and in transit - Integration with Azure AD

Cost: - Hot and Cool storage tiers - Lifecycle management policies - Same pricing as Blob Storage - Pay for what you use

Integration¶

• Azure Synapse Analytics
• Azure Databricks
• Azure HDInsight
• Azure Data Factory
• Power BI

Use Cases¶

• Big data analytics
• Data lake for machine learning
• Data archival with analytics access
• Log and telemetry storage
• Data science workloads

Azure Analysis Services¶

Overview¶

• Enterprise-grade analytics engine
• Tabular data models for BI
• In-memory analytics
• Familiar tools and APIs
• Cloud-based BI semantic layer

Key Features¶

Tabular Models: - Multi-dimensional data models - DAX (Data Analysis Expressions) calculations - Relationships between tables - Calculated columns and measures

Performance: - In-memory data compression - Fast query response times - DirectQuery for real-time data - Hybrid models (import + DirectQuery)

Connectivity: - Connect to multiple data sources - SQL Server, Azure SQL Database - Oracle, Teradata, and more - Data refresh scheduling

Security: - Row-level security (RLS) - Object-level security - Azure AD integration - SSL/TLS encryption

Tiers¶

• Developer: Development and testing
• Basic: Small production workloads
• Standard: Enterprise workloads with high performance

Use Cases¶

• Corporate BI semantic layer
• Self-service analytics
• Departmental reporting
• Excel pivot table data source
• Power BI data models

Analytics and Visualization¶

Power BI¶

Overview¶

• Business analytics and visualization platform
• Interactive dashboards and reports
• Self-service BI capabilities
• Cloud and desktop versions

Components¶

Power BI Desktop: - Windows application for report development - Connect to data sources - Transform and model data - Create visualizations and reports

Power BI Service: - Cloud-based platform - Publish and share reports - Collaborate with teams - Schedule data refreshes - Mobile access

Power BI Mobile: - iOS, Android, Windows apps - Access reports and dashboards - Mobile-optimized views - Offline access

Power BI Report Server: - On-premises reporting solution - Host Power BI reports internally - Hybrid deployment option

Key Capabilities¶

Data Connectivity: - 100+ data source connectors - Import or DirectQuery modes - Composite models - Real-time streaming datasets

Data Transformation: - Power Query for data shaping - Clean and transform data - Merge and append queries - Custom functions

Data Modeling: - Relationships between tables - DAX for calculations - Calculated columns and measures - Time intelligence functions

Visualizations: - 30+ built-in visuals - Custom visuals from marketplace - Interactive filtering and drill-down - Cross-highlighting between visuals

Sharing and Collaboration: - Workspaces for team collaboration - Publish to web (public) - Embed in applications - Row-level security - Sharing with specific users

Integration with Azure¶

• Azure Synapse Analytics: DirectQuery or import
• Azure SQL Database: Real-time dashboards
• Azure Analysis Services: Shared semantic models
• Azure Stream Analytics: Real-time streaming
• Azure Databricks: Advanced analytics results

Licensing¶

• Power BI Free: Individual use, limited sharing
• Power BI Pro: Collaboration and sharing
• Power BI Premium: Dedicated capacity, larger datasets

Use Cases¶

• Executive dashboards
• Sales and marketing analytics
• Financial reporting
• Operational monitoring
• Customer analytics
• Real-time dashboards

Azure Databricks¶

Overview¶

• Apache Spark-based analytics platform
• Collaborative workspace for data engineering and science
• Optimized Spark runtime
• Built on Azure infrastructure
• Unified platform for data, analytics, and AI

Key Features¶

Workspace: - Collaborative notebooks (Python, Scala, R, SQL) - Version control integration (Git) - Shared folders and libraries - Interactive development

Clusters: - Managed Spark clusters - Autoscaling capabilities - Spot instances for cost savings - Multiple runtime versions

Delta Lake: - ACID transactions on data lakes - Schema enforcement and evolution - Time travel (version history) - Unified batch and streaming

MLflow: - Machine learning lifecycle management - Experiment tracking - Model registry - Model deployment

Integration: - Azure Data Lake Storage - Azure Synapse Analytics - Power BI - Azure Machine Learning - Azure DevOps

Common Workflows¶

Data Engineering: - ETL/ELT pipelines - Data lake processing - Data quality checks - Incremental data loads

Data Science: - Exploratory data analysis - Feature engineering - Model training and evaluation - Distributed machine learning

Real-Time Analytics: - Structured streaming - Event processing - Real-time dashboards - Streaming ETL

Use Cases¶

• Big data processing and ETL
• Machine learning and AI
• Real-time analytics
• Data lake management
• Collaborative data science

Azure HDInsight¶

Overview¶

• Managed Hadoop, Spark, and big data clusters
• Open-source analytics service
• Enterprise-level SLAs
• Integration with Azure services

Cluster Types¶

Apache Hadoop: - Batch processing with MapReduce - HDFS for storage - YARN for resource management

Apache Spark: - In-memory processing - Batch and streaming analytics - Machine learning (MLlib)

Apache HBase: - NoSQL database on Hadoop - Real-time read/write access - Billion-row tables

Apache Kafka: - Distributed streaming platform - Real-time data pipelines - Message queuing

Apache Storm: - Real-time stream processing - Low-latency processing - Complex event processing

Interactive Query (LLAP): - Hive with LLAP - Interactive SQL queries - Cached query results

Use Cases¶

• Legacy Hadoop workload migrations
• Open-source big data analytics
• Real-time stream processing
• Data lake processing
• IoT data processing

Comparison: Databricks vs HDInsight vs Synapse¶

Best Practices¶

Data Ingestion¶

1. Choose appropriate ingestion method based on latency requirements
2. Use Event Hubs for high-throughput event streaming
3. Implement retry logic for transient failures
4. Monitor ingestion pipelines for failures and performance
5. Partition data appropriately for parallel processing

Data Processing¶

1. Design idempotent pipelines for reprocessing capability
2. Use incremental loading instead of full refreshes
3. Implement data quality checks early in pipeline
4. Optimize for cost with autoscaling and spot instances
5. Log and monitor all processing steps

Data Storage¶

1. Use data lakes for raw data and data warehouses for processed data
2. Implement data lifecycle policies for cost optimization
3. Partition large datasets for query performance
4. Use appropriate file formats (Parquet/ORC for analytics)
5. Implement security at all layers (network, storage, data)

Analytics and Reporting¶

1. Use aggregated tables for common queries
2. Implement caching for frequently accessed data
3. Design for user concurrency in BI tools
4. Optimize data models in Power BI and Analysis Services
5. Schedule refreshes during off-peak hours

Performance Optimization¶

1. Choose right service tier for workload requirements
2. Implement proper indexing in data warehouses
3. Use columnar storage for analytics workloads
4. Monitor and tune queries regularly
5. Scale resources based on demand

Study Tips¶

Key Concepts to Master¶

• Difference between batch and stream processing
• When to use Event Hubs vs IoT Hub
• Data warehouse vs data lake characteristics
• Azure Synapse Analytics components and capabilities
• Power BI components and workflow
• Integration between different services

Common Scenarios¶

1. Real-time IoT dashboard → IoT Hub → Stream Analytics → Power BI
2. Data warehouse ETL → Data Factory → Synapse Analytics → Power BI
3. Big data analytics → Data Lake Storage → Databricks → Power BI
4. Event processing → Event Hubs → Stream Analytics → SQL Database
5. Machine learning pipeline → Data Factory → Databricks → Azure ML

Exam Focus Areas¶

• Understand when to use batch vs stream processing
• Know the purpose of each analytics service
• Understand data flow in analytics pipelines
• Know basic features of each service
• Be able to design simple analytics solutions
• Understand integration between services

Practice Questions¶

• What service would you use for real-time event ingestion?
• How is a data lake different from a data warehouse?
• What Azure service provides interactive BI dashboards?
• When would you use Stream Analytics vs Data Factory?
• What's the difference between Event Hubs and IoT Hub?

Remember¶

• Batch = schedule, high volume, latency tolerant
• Stream = real-time, continuous, low latency
• Data lakes = raw data, all types, explore later
• Data warehouses = processed data, structured, query now
• Integration is key: Services work together in pipelines