Domain 3: Application Observability and Maintenance (15%)¶

Overview¶

This domain covers monitoring, debugging, and maintaining applications running on Kubernetes. You need to understand health checks (probes), logging, monitoring resource usage, debugging failing applications, and API deprecation handling.

Probes and Health Checks¶

📖 Configure Liveness, Readiness and Startup Probes - Detailed probe configuration 📖 Pod Lifecycle - Container Probes - Probe concepts

Probe Types¶

Probe Mechanisms¶

HTTP GET Probe:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
    httpHeaders:
      - name: Custom-Header
        value: "probe-check"
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
  successThreshold: 1

TCP Socket Probe:

readinessProbe:
  tcpSocket:
    port: 3306
  initialDelaySeconds: 5
  periodSeconds: 10

Exec Probe:

livenessProbe:
  exec:
    command:
      - cat
      - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

gRPC Probe:

livenessProbe:
  grpc:
    port: 50051
    service: my-service
  initialDelaySeconds: 10
  periodSeconds: 10

Probe Parameters¶

Complete Probe Example¶

apiVersion: v1
kind: Pod
metadata:
  name: probed-app
spec:
  containers:
    - name: app
      image: myapp:1.0
      ports:
        - containerPort: 8080
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        failureThreshold: 30
        periodSeconds: 10
        # Allows up to 300 seconds (30 * 10) for startup
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 3

When to Use Each Probe¶

Liveness Probe: - Application can get into a broken state that only a restart can fix - Deadlocked processes - Internal corruption - Caution: Overly aggressive liveness probes can cause restart loops

Readiness Probe: - Application needs time to load data or warm up - Application depends on external services - Application should temporarily stop receiving traffic during maintenance - Connection pool exhaustion

Startup Probe: - Application has a slow or variable startup time - Prevents liveness probe from killing the container during startup - Legacy applications with long initialization

Container Logging¶

📖 Logging Architecture - Kubernetes logging concepts

kubectl logs Commands¶

# Basic log viewing
kubectl logs <pod-name>

# Specific container in a multi-container Pod
kubectl logs <pod-name> -c <container-name>

# Stream logs in real time
kubectl logs <pod-name> -f

# Previous container instance (after a restart)
kubectl logs <pod-name> --previous

# Last N lines
kubectl logs <pod-name> --tail=50

# Logs since a duration
kubectl logs <pod-name> --since=1h
kubectl logs <pod-name> --since=30m

# Logs since a specific time
kubectl logs <pod-name> --since-time="2024-01-15T10:00:00Z"

# All containers in the Pod
kubectl logs <pod-name> --all-containers=true

# Logs from all Pods matching a label
kubectl logs -l app=myapp

# Logs from all Pods matching a label with follow
kubectl logs -l app=myapp -f --max-log-requests=10

# Combine flags
kubectl logs <pod-name> -c sidecar --tail=100 --since=30m

Logging Best Practices for CKAD¶

• Applications should log to stdout/stderr, not to files
• Use structured logging (JSON) when possible
• Include timestamps, log levels, and context in log messages
• For file-based logs, use a sidecar container pattern to ship logs

Monitoring and Resource Usage¶

📖 Resource Metrics Pipeline - Metrics pipeline 📖 Monitoring Resource Usage - Resource monitoring

kubectl top Commands¶

# Node resource usage
kubectl top nodes

# Pod resource usage
kubectl top pods
kubectl top pods -n <namespace>
kubectl top pods --all-namespaces
kubectl top pods -l app=myapp

# Per-container resource usage
kubectl top pod <pod-name> --containers

# Sort by CPU or memory
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory

Note: kubectl top requires the Metrics Server to be installed in the cluster.

Understanding Resource Units¶

• CPU: Measured in millicores (m). 1 CPU = 1000m. 100m = 0.1 CPU.
• Memory: Measured in bytes. Common units: Ki (kibibytes), Mi (mebibytes), Gi (gibibytes).

Debugging Applications¶

📖 Debug Pods - Pod troubleshooting 📖 Debug Services - Service troubleshooting 📖 Debug Running Pods - Inspecting running containers 📖 Get a Shell to a Running Container - Exec into containers

Debugging Workflow¶

Step 1: Check Pod Status

kubectl get pods -n <namespace>
kubectl get pods -o wide    # Shows node and IP

Common Pod statuses: | Status | Meaning | |--------|---------| | Pending | Cannot be scheduled (insufficient resources, node affinity) | | ContainerCreating | Image being pulled or volume being mounted | | Running | All containers started | | CrashLoopBackOff | Container keeps crashing and being restarted | | ImagePullBackOff | Cannot pull the container image | | ErrImagePull | Failed to pull the container image | | OOMKilled | Container exceeded its memory limit | | Completed | Container finished successfully (for Jobs) |

Step 2: Describe the Pod

kubectl describe pod <pod-name> -n <namespace>

Step 3: Check Logs

kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -c <container> --previous    # After crash

Step 4: Exec Into the Container

kubectl exec -it <pod-name> -- /bin/sh
kubectl exec -it <pod-name> -c <container> -- /bin/bash

# Run a specific command
kubectl exec <pod-name> -- env
kubectl exec <pod-name> -- cat /etc/config/app.conf
kubectl exec <pod-name> -- wget -qO- http://localhost:8080/healthz

Step 5: Check Events

kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp
kubectl get events --field-selector reason=Failed

Common Issues and Fixes¶

CrashLoopBackOff: - Check logs for application errors - Verify the command/args in the container spec - Check if the application needs environment variables or config - Verify resource limits are sufficient

ImagePullBackOff: - Check the image name and tag - Verify the image exists in the registry - Check imagePullSecrets for private registries - Verify network connectivity to the registry

Pending: - Check node resources (kubectl describe node) - Verify node selectors and tolerations - Check PVC binding (if using persistent volumes) - Check ResourceQuotas in the namespace

Service not routing traffic: - Verify Service selector matches Pod labels - Check endpoints: kubectl get endpoints <service-name> - Verify target port matches the container port - Check NetworkPolicies

API Deprecations¶

📖 Kubernetes Deprecation Policy - Deprecation rules 📖 Deprecated API Migration Guide - API migration paths

API Version Lifecycle¶

1. Alpha (v1alpha1): Experimental, may change or be removed without notice
2. Beta (v1beta1): Feature is well-tested, enabled by default, may have minor changes
3. Stable (v1): Feature is stable, API will not change in backward-incompatible ways

Checking Available APIs¶

# List all API resources
kubectl api-resources

# List API versions
kubectl api-versions

# Check if a specific resource is available
kubectl api-resources | grep networkpolicies

# Explain a resource (shows API version and fields)
kubectl explain deployment
kubectl explain deployment.spec.strategy

Notable API Changes¶

• Ingress: Moved from extensions/v1beta1 and networking.k8s.io/v1beta1 to networking.k8s.io/v1
• NetworkPolicy: Moved from extensions/v1beta1 to networking.k8s.io/v1
• CronJob: Moved from batch/v1beta1 to batch/v1

Best Practice¶

Always use the latest stable API version. Check kubectl explain <resource> to see the current API version for any resource.