Azure AI Services Overview¶

Table of Contents¶

1. Azure AI Services Introduction
2. Computer Vision Services
3. Language Services
4. Speech Services
5. Decision Services
6. Azure OpenAI Service
7. Service Integration and Architecture
8. Exam Tips

Azure AI Services Introduction¶

Overview¶

Azure AI Services (formerly Cognitive Services) are cloud-based AI capabilities that enable developers to build intelligent applications without deep AI/ML expertise.

Key Categories¶

1. Vision - Analyze images and videos
2. Language - Understand and analyze text
3. Speech - Convert speech to text and vice versa
4. Decision - Make intelligent recommendations and detect anomalies
5. Azure OpenAI - Access to advanced language models

Common Features¶

• REST APIs - Simple HTTP-based interfaces
• SDKs - Available for .NET, Python, Java, JavaScript, Go
• Containers - Deploy on-premises or edge
• Custom Models - Train with your own data
• Multi-region - Global availability
• Security - Azure AD integration, key-based auth

Service Tiers¶

• Free Tier - Limited transactions for testing
• Standard Tier - Pay-as-you-go pricing
• Commitment Tier - Discounted rates for committed usage

Computer Vision Services¶

Azure Computer Vision¶

Analyzes images and extracts information.

Key Capabilities¶

1. Image Analysis
2. Object detection
3. Scene understanding
4. Brand detection
5. Face detection (not identification)
6. Color scheme extraction

7. Adult/racy content detection

8. Optical Character Recognition (OCR)

9. Read API for text extraction
10. Handwritten text support
11. Multiple language support

12. Document analysis

13. Spatial Analysis

14. People counting
15. Social distancing monitoring
16. Zone occupancy detection

Common Use Cases¶

• Catalog search by image
• Accessibility (image descriptions)
• Content moderation
• Document digitization
• Retail analytics

API Features¶

POST {endpoint}/vision/v3.2/analyze?visualFeatures=Categories,Tags,Description
Content-Type: application/json

{
  "url": "https://example.com/image.jpg"
}

Custom Vision¶

Build custom image classification and object detection models.

Project Types¶

1. Classification
2. Multi-class (single tag per image)

3. Multi-label (multiple tags per image)

4. Object Detection

5. Identify and locate objects with bounding boxes

Training Process¶

1. Create project
2. Upload and tag images (minimum 5-15 per tag)
3. Train model
4. Evaluate performance (precision, recall)
5. Publish prediction endpoint
6. Iterate and improve

Performance Metrics¶

• Precision - Accuracy of predictions
• Recall - Coverage of correct items
• AP (Average Precision) - Overall model quality

Face Service¶

Detect, recognize, and analyze human faces.

Capabilities¶

1. Face Detection
2. Location (bounding box)
3. Landmarks (eyes, nose, mouth)

4. Attributes (age, emotion, glasses, etc.)

5. Face Verification

6. Compare two faces for similarity

7. Face Identification

8. Match face against person group

9. Requires training data

10. Face Grouping

11. Organize unknown faces by similarity

Limited Access Features¶

• Face identification and verification require approval
• Complies with Responsible AI principles
• Use cases must be reviewed by Microsoft

Best Practices¶

• Use well-lit, frontal images
• Minimum face size: 36x36 pixels
• Maximum 1000 faces per image
• Store face IDs, not original images

Form Recognizer (Document Intelligence)¶

Extract information from documents.

Model Types¶

1. Prebuilt Models
2. Invoices
3. Receipts
4. Business cards
5. ID documents

6. W-2 forms

7. Custom Models

8. Structured forms

9. Unstructured documents

10. General Document Model

11. Layout analysis
12. Key-value pairs
13. Tables

Key Features¶

• Table extraction
• Key-value pair detection
• Handwriting support
• Multi-page document support
• Confidence scores

Language Services¶

Azure AI Language¶

Comprehensive text analysis and NLP capabilities.

Core Features¶

1. Named Entity Recognition (NER)
2. Person, location, organization
3. Date, time, quantity
4. Email, URL, phone number

5. Custom entity recognition

6. Key Phrase Extraction

7. Main talking points
8. Important concepts

9. Works across 120+ languages

10. Sentiment Analysis

11. Positive, negative, neutral, mixed
12. Document and sentence-level analysis

13. Opinion mining (aspect-based sentiment)

14. Language Detection

15. Identify language of text
16. Confidence scores

17. Supports 120+ languages

18. Entity Linking

19. Disambiguate entities
20. Link to Wikipedia
21. Knowledge base integration

Advanced Features¶

1. Text Analytics for Health
2. Medical entity extraction
3. UMLS/SNOMED/ICD-10 coding

4. Relation extraction

5. Personally Identifiable Information (PII) Detection

6. Redact sensitive information
7. HIPAA and GDPR compliance

8. Custom PII categories

9. Custom Text Classification

10. Single-label classification
11. Multi-label classification

12. Minimum 10 documents per class

13. Custom Named Entity Recognition

14. Domain-specific entities
15. Minimum 10 labeled documents

Question Answering¶

Build FAQ and conversational Q&A systems.

Components: - Knowledge base (Q&A pairs) - Multiple sources (URLs, files, editorial) - Active learning - Multi-turn conversations - Synonyms and alterations

Development Process: 1. Create knowledge base 2. Add Q&A pairs from sources 3. Train and test 4. Publish endpoint 5. Improve with active learning

Conversational Language Understanding (CLU)¶

Build custom natural language understanding models.

Concepts: - Intents - User's goal (e.g., BookFlight) - Entities - Key information (e.g., destination, date) - Utterances - Example phrases

Development: 1. Define intents and entities 2. Add training utterances (10-15 per intent) 3. Train model 4. Test and evaluate 5. Deploy to prediction endpoint

Azure AI Translator¶

Real-time text translation.

Features¶

• 100+ languages
• Document translation
• Custom translation models
• Transliteration
• Language detection
• Dictionary lookup
• Batch translation

Translation Options¶

1. Text Translation API
2. Real-time translation
3. Multiple target languages

4. Custom models

5. Document Translation

6. Preserve formatting
7. Batch processing

8. Multiple file formats

9. Custom Translator

10. Domain-specific translation
11. Terminology management
12. Translation memory

Speech Services¶

Azure AI Speech¶

Convert speech to text and text to speech.

Speech-to-Text (STT)¶

Capabilities: - Real-time transcription - Batch transcription - Multiple languages (100+) - Custom speech models - Profanity filtering - Diarization (speaker identification)

Recognition Modes: - Interactive (short commands) - Conversation (dialog) - Dictation (continuous speech)

Custom Speech: - Acoustic models (audio quality) - Language models (domain vocabulary) - Pronunciation models

Text-to-Speech (TTS)¶

Features: - Neural voices (natural sounding) - 400+ voices across 140+ languages - SSML (Speech Synthesis Markup Language) - Custom neural voice - Emotional styles - Speaking styles (newscast, cheerful, etc.)

SSML Elements:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
    <voice name="en-US-JennyNeural">
        <prosody rate="medium" pitch="high">
            Hello, how can I help you today?
        </prosody>
    </voice>
</speak>

Speech Translation¶

Capabilities: - Real-time speech translation - 30+ languages - Multiple target languages simultaneously - Translation to text or speech

Speaker Recognition¶

Features: - Text-dependent verification - Text-independent verification - Speaker identification

Decision Services¶

Azure AI Personalizer¶

Deliver personalized experiences using reinforcement learning.

How It Works¶

1. Context - Information about current situation
2. Actions - Available options to choose from
3. Rank - Service ranks actions
4. Reward - Feedback on chosen action
5. Learn - Model improves over time

Use Cases¶

• Content recommendations
• Product suggestions
• Ad placement
• Personalized UI

Key Concepts¶

• Features - Contextual and action characteristics
• Exploration - Try different options
• Exploitation - Use best-known option
• Apprentice Mode - Shadow existing system

Anomaly Detector¶

Identify anomalies in time-series data.

Detection Modes¶

1. Batch Detection
2. Analyze entire dataset

3. Detect anomalies across all points

4. Streaming Detection

5. Real-time anomaly detection

6. Latest data point analysis

7. Multivariate Anomaly Detection

8. Analyze multiple related metrics
9. Detect complex patterns

Use Cases¶

• IoT sensor monitoring
• Business metrics monitoring
• Fraud detection
• System health monitoring

Configuration¶

• Sensitivity - Detection threshold (0-99)
• Custom Interval - Data point frequency
• Seasonality - Periodic patterns

Content Moderator (Being deprecated)¶

Detect potentially offensive content.

Note: Being replaced by Azure AI Content Safety.

Azure AI Content Safety¶

Features: - Text moderation - Image moderation - Multi-category detection (hate, violence, sexual, self-harm) - Severity levels (0-6) - Blocklist management - Custom categories

Azure OpenAI Service¶

Overview¶

Access to OpenAI's advanced language models through Azure.

Available Models¶

1. GPT-4 - Most capable, multimodal
2. GPT-3.5 - Fast, cost-effective
3. Embeddings - Text vectorization
4. DALL-E - Image generation
5. Whisper - Speech recognition

Key Features¶

• Enterprise-grade security
• Private networking
• Managed infrastructure
• Content filtering
• Responsible AI tools

Common Patterns¶

1. Completions - Generate text
2. Chat - Conversational AI
3. Embeddings - Semantic search
4. Fine-tuning - Custom models

Use Cases¶

• Content generation
• Code assistance
• Summarization
• Translation
• Question answering
• Chatbots

Service Integration and Architecture¶

Multi-Service Resource¶

• Single endpoint for multiple services
• Unified key management
• Consolidated billing
• Simplified deployment

Single-Service Resource¶

• Dedicated endpoint per service
• Granular access control
• Separate cost tracking
• Region-specific deployment

Authentication Methods¶

1. Subscription Keys
2. Primary and secondary keys
3. Rotate without downtime

4. Simple to implement

5. Azure Active Directory

6. Identity-based access
7. Managed identities
8. Role-based access control (RBAC)

SDKs and APIs¶

Available SDKs: - .NET (C#) - Python - Java - JavaScript/Node.js - Go

REST API Format:

POST https://{endpoint}/cognitive/service/v1.0/analyze
Ocp-Apim-Subscription-Key: {your-key}
Content-Type: application/json

{
  "parameter": "value"
}

Container Deployment¶

Benefits: - On-premises deployment - Edge computing - Data sovereignty - Disconnected scenarios

Requirements: - Docker runtime - Internet for billing - Minimum resource allocation - License acceptance

Best Practices¶

1. Security
2. Use Azure AD when possible
3. Rotate keys regularly
4. Use private endpoints

5. Enable logging and monitoring

6. Performance

7. Choose appropriate region
8. Implement retry logic
9. Cache results when appropriate

10. Use batch operations

11. Cost Optimization

12. Use free tier for development
13. Choose right service tier
14. Implement commitment pricing

15. Monitor usage

16. Reliability

17. Handle throttling (429 errors)
18. Implement circuit breakers
19. Use health checks
20. Plan for failover

Exam Tips¶

Key Concepts to Remember¶

1. Service Selection
2. Know which service for which scenario
3. Understand service capabilities

4. Differentiate between similar services

5. Vision Services

6. Computer Vision vs Custom Vision vs Face
7. OCR capabilities and limitations

8. Form Recognizer prebuilt models

9. Language Services

10. Sentiment analysis vs opinion mining
11. Question Answering vs CLU

12. When to use custom models

13. Speech Services

14. STT vs TTS vs Translation
15. Neural voices vs standard voices

16. Custom speech scenarios

17. Decision Services

18. Personalizer reinforcement learning
19. Anomaly Detector modes

20. Content Safety categories

21. Authentication

22. Subscription keys vs Azure AD
23. When to use managed identities

24. RBAC roles

25. Deployment

26. Container deployment scenarios
27. Multi-service vs single-service
28. Region selection considerations

Common Exam Scenarios¶

1. Content Moderation
2. Use Azure AI Content Safety
3. Configure severity thresholds

4. Implement blocklists

5. Document Processing

6. Form Recognizer for structured data
7. Computer Vision OCR for simple text

8. Choose prebuilt vs custom models

9. Language Understanding

10. CLU for custom intents/entities
11. Question Answering for FAQ

12. Translator for multi-language

13. Voice Applications

14. Speech-to-Text for transcription
15. Text-to-Speech for voice output

16. Speech Translation for real-time

17. Personalization

18. Personalizer for dynamic content
19. Context + Actions + Reward pattern
20. Apprentice mode for learning

Important Limits and Quotas¶

Pricing Considerations¶

• Free tier for development and testing
• Standard tier pay-per-use
• Commitment tiers for predictable workloads
• Custom models may have additional costs
• Container deployment requires connected billing

Study Focus Areas¶

1. Service Capabilities (30%)
2. What each service does
3. Key features and limitations

4. When to use which service

5. Implementation (25%)

6. REST API usage
7. SDK implementation

8. Authentication methods

9. Custom Models (20%)

10. Training requirements
11. Data preparation

12. Model evaluation

13. Deployment (15%)

14. Container scenarios
15. Resource types

16. Security configuration

17. Responsible AI (10%)

18. Content filtering
19. Limited access features
20. Compliance requirements

Quick Reference¶

Computer Vision: - Analyze: Tags, objects, faces, OCR - Custom Vision: Classification & object detection - Face: Detect, verify, identify (limited access)

Language: - Analyze: Sentiment, entities, key phrases - Understand: CLU for intents/entities - Q&A: Question Answering service - Translate: Text and documents

Speech: - Recognize: Speech-to-Text - Synthesize: Text-to-Speech - Translate: Speech Translation - Identify: Speaker Recognition

Decision: - Personalize: Reinforcement learning recommendations - Detect: Time-series anomaly detection - Moderate: Content Safety

Azure OpenAI: - Generate: GPT models - Embed: Similarity and search - Create: DALL-E images - Transcribe: Whisper speech

Final Exam Tips¶

1. Read questions carefully - focus on requirements
2. Eliminate obviously wrong answers first
3. Consider cost, performance, and complexity
4. Remember responsible AI principles
5. Think about real-world scenarios
6. Don't overthink - trust your preparation
7. Manage time - don't get stuck on one question
8. Review flagged questions if time permits