Skip to content

Scrape Prometheus metrics from multiple containers in a Pod

Imported from Confluence

Content may be outdated. Verify before following any procedures. View original | Last updated: September 2022

Scrape config:

# Scrape config to crape metrics from more that one container in a Pod
# * `prometheus.io/scrape-container`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. This
#    will be the same for every container in the pod that is scraped.
# * this will scrape every container in a pod with `prometheus.io/scrape-container` set to true and the
#   port with name `metrics` exists in the container
# * note `prometheus.io/port` is no longer honored. You must name the port(s) to scrape `metrics`
- honor_labels: true
  job_name: 'kubernetes-pods-container'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_container]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: metrics
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

Example annotations:

    prometheus.io/path: /stats/prometheus
    prometheus.io/scrape-container: "true"

Example ports:

spec:
  containers:
  - name: container1
    ports:
    - containerPort: 3000
      name: metrics
      protocol: TCP
  - name: container2
    ports:
    - containerPort: 3001
      name: metrics
      protocol: TCP