05 - Operators, OperatorHub, Monitoring, Troubleshooting¶

Operators and OLM¶

Operators are Kubernetes-native software that packages, deploys, and manages applications via Custom Resources. Operator Lifecycle Manager (OLM) is the OpenShift component that manages operator install / upgrade / removal.

Key resources¶

Inspect catalog¶

oc get packagemanifest -n openshift-marketplace
oc get packagemanifest <name> -n openshift-marketplace -o yaml | head -100

oc describe packagemanifest openshift-pipelines-operator-rh -n openshift-marketplace

Install operator (CLI)¶

1. Create OperatorGroup (if installing into a custom namespace):

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: pipeline-operators
  namespace: openshift-pipelines
spec:
  targetNamespaces:
  - openshift-pipelines

(For cluster-wide operators installed in openshift-operators, an OperatorGroup already exists.)

1. Create Subscription:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-pipelines-operator
  namespace: openshift-operators
spec:
  channel: latest
  name: openshift-pipelines-operator-rh
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

1. Apply and wait:

oc apply -f operatorgroup.yaml
oc apply -f subscription.yaml

oc get csv -A
oc get subscription -A
oc get installplan -A

When the CSV phase reaches Succeeded, the operator is ready.

Install operator (web console)¶

OperatorHub → search → install → choose channel → install. Same effect as the YAML above.

Manual approval (sometimes asked)¶

Set installPlanApproval: Manual. After the InstallPlan is created, approve manually:

oc edit installplan -n <namespace>
# Set spec.approved: true

Update / upgrade operator¶

Change channel:

oc patch subscription <name> -n openshift-operators \
    --type=merge \
    -p '{"spec":{"channel":"stable"}}'

OLM creates a new InstallPlan with the upgrade.

Uninstall operator¶

oc delete subscription <name> -n openshift-operators
oc delete csv <csv-name> -n openshift-operators
# Delete CRDs too if you want to fully remove (be careful):
oc get crds | grep <operator-keyword>
oc delete crd <crd-name>

Cluster monitoring¶

OpenShift's built-in monitoring stack lives in openshift-monitoring:

• Prometheus - metrics collection
• Alertmanager - alert routing
• Grafana (deprecated in newer versions; use console dashboards instead)
• node-exporter - per-node metrics
• kube-state-metrics - K8s object state metrics
• prometheus-operator - manages it all

oc -n openshift-monitoring get pods
oc -n openshift-monitoring get routes prometheus-k8s
oc -n openshift-monitoring get routes alertmanager-main

Cluster monitoring config¶

Edit the cluster-monitoring-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        spec:
          storageClassName: gp3-csi
          resources:
            requests:
              storage: 50Gi
    alertmanagerMain:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3-csi
          resources:
            requests:
              storage: 10Gi

oc apply -f cluster-monitoring-config.yaml

User workload monitoring¶

When enableUserWorkload: true, a separate Prometheus runs in openshift-user-workload-monitoring to scrape user-defined ServiceMonitor resources.

oc -n openshift-user-workload-monitoring get pods

# User creates ServiceMonitor in their own project

Alerts¶

oc get prometheusrule -A             # all alert rules
oc -n openshift-monitoring get prometheusrule

User-defined alert example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-alerts
  namespace: myapp
spec:
  groups:
  - name: app
    rules:
    - alert: HighErrorRate
      expr: rate(http_errors_total[5m]) > 0.1
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: 'High error rate'

Logging¶

OpenShift Logging is an operator-installed stack (LokiStack on newer versions, EFK on older):

oc -n openshift-logging get pods

Install via OperatorHub: "Red Hat OpenShift Logging" + "Loki Operator".

For exam purposes, you probably won't deploy logging from scratch but should know the resource: ClusterLogForwarder for sending logs to external systems.

Troubleshooting¶

General workflow¶

1. oc get co - any cluster operator degraded?
2. oc get nodes - all Ready?
3. oc get pods -A | grep -v Running | grep -v Completed
4. oc describe <bad-pod> - look at Events
5. oc logs <pod> -c <container> (and -p for previous container)
6. oc adm must-gather - capture cluster diagnostic

must-gather¶

# Cluster-wide
oc adm must-gather

# Operator-specific
oc adm must-gather --image=registry.redhat.io/openshift-logging/cluster-logging-rhel8-operator:latest

# Output goes to a directory in current working dir.
# Tarball it for support:
tar -czf must-gather-$(date +%F).tar.gz must-gather.local.*

Pod debugging¶

oc describe pod <name>                      # events
oc logs <pod>
oc logs -p <pod>                            # previous (crashed) container
oc logs <pod> -c <container-name>
oc rsh <pod>                                # shell into running container
oc debug pod/<pod>                          # spawn debug pod
oc debug deploy/<dep>                       # debug from deployment
oc debug node/<nodename>                    # privileged pod on node

Network debugging¶

oc debug node/<node>
# inside:
chroot /host
ip a
ip r
iptables -L
crictl ps

In a pod:

oc rsh <pod>
nslookup <service>
curl <service>
ping <pod-ip>

Common error messages¶

Common operator-related exam tasks¶

Install an operator (e.g., Pipelines, Serverless, GitOps)¶

Apply the Subscription YAML in the right namespace and wait for the CSV.

Verify operator install¶

oc get csv -n openshift-operators
oc get subscription -n openshift-operators
oc get pods -n openshift-operators

Configure user workload monitoring¶

Apply cluster-monitoring-config ConfigMap with enableUserWorkload: true.

Install a logging stack¶

Install Logging Operator + Loki Operator from OperatorHub. Create a ClusterLogging instance.

Capture diagnostic for support¶

oc adm must-gather
tar -czf must-gather.tar.gz must-gather.local.*

Verification¶

After operator / monitoring changes:

• oc get csv -A shows operators in Succeeded phase
• oc get pods -n <operator-namespace> shows operator pods Running
• oc -n openshift-monitoring get pods shows monitoring pods Running
• For user workload monitoring: oc -n openshift-user-workload-monitoring get pods shows pods Running
• The web console's monitoring views show metrics and alerts