Fleet Management¶

📖 GPU Operator Documentation - GPU lifecycle management

Multi-Cluster GPU Operations¶

Fleet Architecture¶

Centralized Management: - Single pane of glass for all GPU clusters - Consistent policies across environments - Centralized monitoring and alerting - Unified reporting and compliance

Cluster Types: - Training clusters - large GPU pools for model development - Inference clusters - optimized for serving workloads - Development clusters - shared resources for experimentation - Edge clusters - GPU nodes at edge locations

Inventory Management¶

• Track GPU models, counts, and locations across all clusters
• Monitor GPU lifecycle (age, warranty status, utilization history)
• Capacity tracking and trending
• Asset management integration

Driver and Firmware Lifecycle¶

NVIDIA Driver Management¶

Driver Branches: - Production branch - stable, long-term support - New feature branch - latest features, shorter support - Long-lived branch - extended support for specific deployments - 📖 Driver Documentation - Data center drivers

Update Process: 1. Test new driver on non-production nodes 2. Validate with representative workloads 3. Update canary nodes in production 4. Monitor for issues over confidence period 5. Rolling update to remaining nodes 6. Keep previous driver available for rollback

Kubernetes Driver Updates (GPU Operator): - GPU Operator manages driver lifecycle automatically - Rolling updates with node drain - Driver version pinning via operator configuration - Automatic driver installation on new nodes

Firmware Updates¶

GPU Firmware (InfoROM, VBIOS): - Less frequent than driver updates - Requires GPU reset or node reboot - Schedule during maintenance windows - Validate compatibility with driver version - nvidia-smi reports firmware versions

NIC Firmware: - ConnectX adapter firmware updates - Managed via Mellanox Firmware Tools (MFT) - Coordinate with InfiniBand fabric updates - Test network performance after update

Compatibility Matrix¶

Always verify compatibility between: - GPU driver version - CUDA toolkit version - GPU firmware version - Container toolkit version - Kubernetes version - GPU Operator version

Configuration Management¶

Infrastructure as Code¶

Ansible for GPU Nodes: - GPU driver installation and configuration - DCGM agent deployment - Network configuration (InfiniBand, GPUDirect) - Storage mount configuration - Security hardening

Terraform for GPU Infrastructure: - Cloud GPU instance provisioning - Network and storage infrastructure - Kubernetes cluster creation - Load balancer configuration

Helm for Kubernetes GPU Stack: - GPU Operator deployment - Monitoring stack (Prometheus, Grafana, DCGM Exporter) - Inference servers (Triton, NIM) - Autoscaler configuration

Configuration Consistency¶

Golden Image Management: - Base OS image with GPU drivers pre-installed - Container images with verified CUDA versions - Consistent across all environments - Automated image building and testing

Drift Detection: - Monitor nodes for configuration drift - Alert on driver version mismatches - Detect unauthorized software changes - Automated remediation for known drift patterns

Security and Compliance¶

GPU Security¶

Driver Security: - Apply security patches promptly - Monitor NVIDIA security bulletins - CVE tracking and remediation - Minimize attack surface (disable unused features)

Container Security: - Scan NGC container images for vulnerabilities - Use signed and verified images - Runtime security monitoring - GPU device access control

Network Security: - InfiniBand fabric isolation - GPUDirect RDMA access control - Network segmentation for multi-tenant environments - Encrypted communication where required

Compliance¶

Audit Logging: - GPU resource access and allocation - Driver and firmware changes - Configuration modifications - User and job activity

Reporting: - GPU utilization reports for capacity planning - Cost allocation per team/project - Compliance posture dashboards - Security vulnerability status

Capacity Planning¶

Utilization Analysis¶

• Track GPU utilization trends over time
• Identify peak usage periods
• Measure queue wait times for jobs
• Calculate effective GPU utilization vs allocated

Growth Planning¶

• Project GPU demand based on team growth
• Account for new model sizes and workloads
• Lead time for hardware procurement (months)
• Budget planning for GPU infrastructure

Optimization¶

• Identify underutilized GPUs and reallocate
• Right-size GPU allocations per workload
• MIG for inference workload consolidation
• Time-slicing for development environments
• Spot/preemptible instances for batch workloads

Key Exam Concepts¶

• Multi-cluster management architecture
• Driver update strategies and rollback procedures
• Compatibility matrix management (driver, CUDA, firmware)
• Infrastructure as Code for GPU environments
• Security patching and compliance requirements
• Capacity planning and utilization optimization