A step-by-step guide to implement metrics for your ad-hoc Operator in Golang by Prometheus on Kubernetes

import (
"github.com/prometheus/client_golang/prometheus"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)

var (
goobers = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "goobers_total",
Help: "Number of goobers proccessed",
},
)
gooberFailures = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "goober_failures_total",
Help: "Number of failed goobers",
},
)
)

func initmetrics() {
// Register custom metrics with the global prometheus registry
metrics.Registry.MustRegister(goobers, gooberFailures)
}
Courtesy to “kubebuilder” site

Step 1-Prepare a cluster

Step 2-Write and Deploy your Operator on your Cluster

https://github.com/IBM/operator-sample-go

Step 3-Deploy Prometheus on your cluster

  1. Go to https://operatorhub.io/
  2. Search for Prometheus
  3. Click on “Prometheus Operator”
  4. Then click on “Install” button and follow the instructions.
  5. In brief what you need to do is;
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.20.0/install.sh | bash -s v0.20.0kubectl create -f https://operatorhub.io/install/prometheus.yaml
Operatorhub.io page
Installation instructions

Step 4-Make the Prometheus UI accessible on the cluster

apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector: {}
serviceMonitorNamespaceSelector: {}
resources:
requests:
memory: 400Mi
enableAdminAPI: true
apiVersion: v1
kind: Service
metadata:
name: prometheus
annotations:
service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: "public"
spec:
type: LoadBalancer
ports:
- name: web
port: 9090
protocol: TCP
targetPort: 9090
selector:
prometheus: prometheus
apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
type: NodePort
ports:
- name: web
nodePort: 30901
port: 9090
protocol: TCP
targetPort: web
selector:
prometheus: prometheus
From the original version# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus

To

# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
- ../prometheus
make generatemake manifests
# Prometheus Monitor Service (Metrics)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
control-plane: controller-manager
name: controller-manager-metrics-monitor
namespace: system
spec:
endpoints:
- path: /metrics
port: https
scheme: https
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
control-plane: controller-manager
Screen capture courtesy to Thomas Südbröcker’s blog

Step 5-Implement a function to gather the Operator metrics

package controllers

import (
"github.com/prometheus/client_golang/prometheus"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)

var (
goobers = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "goobers_total",
Help: "Number of goobers proccessed",
},
)
gooberFailures = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "goober_failures_total",
Help: "Number of failed goobers",
},
)
)

func init() {
// Register custom metrics with the global prometheus registry
metrics.Registry.MustRegister(goobers, gooberFailures)
}

...
// Add metrics information
initmetrics()
goobers.Inc()
...
Diagram courtesy to Thomas Südbröcker’s who is a big fan of diagrams :D and is the diagram guru of the team

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store