how to use blackbox_exporter inside kubernetes

[ganting]

my rc yaml:

apiVersion: v1
kind: ReplicationController
metadata:
  name: blackbox-exporter
spec:
  replicas: 1
  selector:
    app: blackbox-exporter
  template:
    metadata:
      name: blackbox-exporter
      labels:
        app: blackbox-exporter
    spec:
      containers:
      - image: ****/prometheus/blackbox-exporter:1.0
        name: blackbox
        ports:
        - containerPort: 9115
          name: blackbox

my svc yaml:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  labels:
    name: rabbitmq
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/probe: "true"
spec:
  ports:
    - port: 9090
      name: exporter
      targetPort: exporter
      protocol: TCP
  selector:
    app: rabbitmq

but prometheus targets found down of probe

[brian-brazil]

This is probably a DNS issue on your end.

[ganting]

i split the svc yaml to two files
one for rabbitmq-exporter:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  labels:
    name: rabbitmq
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
spec:
  ports:
    - port: 9090
      name: exporter
      targetPort: exporter
      protocol: TCP
  selector:
    app: rabbitmq

the other for blackbox-exporter, service name is blackbox and service port is 80

apiVersion: v1
kind: Service
metadata:
  name: blackbox
  labels:
    name: blackbox
  annotations:
    prometheus.io/probe: "true"
spec:
  ports:
    - port: 80
      targetPort: blackbox
      protocol: TCP
  selector:
    rabbitmq-blackbox: "true"

it's work right for prometheus

my rc for rabbitmq is:

apiVersion: v1
kind: ReplicationController
metadata:
  name: rabbitmq
spec:
  replicas: 1
  selector:
    app: rabbitmq
    rabbitmq-blackbox: "true"
  template:
    metadata:
      name: rabbitmq
      labels:
        app: rabbitmq
        rabbitmq-blackbox: "true"
    spec:
      containers:
      - image: docker.hikvision.com.cn/prometheus/rabbitmq:3.5.4-management
        imagePullPolicy: Always
        name: rabbitmq
        ports:
        - containerPort: 5672
          name: service
        - containerPort: 15672
          name: management
      - image: docker.hikvision.com.cn/prometheus/blackbox-exporter:1.0
        imagePullPolicy: Always
        name: rabbitmq-blackbox
        ports:
        - containerPort: 9115
          name: blackbox
      - image: docker.hikvision.com.cn/prometheus/rabbitmq-exporter:1.0
        imagePullPolicy: Always
        name: rabbitmq-exporter
        ports:
        - containerPort: 9090
          name: exporter

my prometheus.yml for probe is:

- job_name: 'kubernetes-services'

  metrics_path: /probe
  params:
    module: [http_2xx]

  tls_config:
    #ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default.svc'
    in_cluster: true

  relabel_configs:
  - source_labels: [__meta_kubernetes_role, __meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: service;true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_role]
    target_label: kubernetes_role
  - source_labels: [__meta_kubernetes_service_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name

my questions is:

• why service name must be blackbox, and why service port must be 80?
• does blackbox-exporter have to be written in rabbitmq rc yaml or not?
• if i have another service like redis use blackbox-exporter, how to do?

thanks

[jimmidyson]

As @brian-brazil check that DNS is working properly for you in your pods.

Secondly, you need to annotate the service you want to be monitored via the blackbox exporter, not the blackbox exporter, with prometheus.io/probe=true. Your Prometheus configuration above will then use the blackbox exporter to monitor any services that are annotated correctly.

[jimmidyson]

why service name must be blackbox, and why service port must be 80?

It doesn't you just need to make sure that you are setting the __address__ label during relabelling to point to the address that the blackbox exporter is available at, you can choose whatever you want for that as long as it is resolvable/routeable in Prometheus.

does blackbox-exporter have to be written in rabbitmq rc yaml or not?

It does not.

if i have another service like redis use blackbox-exporter, how to do?

Just annotate the redis service with prometheus.io/probe=true

[ganting]

thanks, i solved my problem by using your method.

[jimmidyson]

Excellent!