Scraping Prometheus metrics with Nomad and Consul Connect

Exposing the metrics outside your task container
Scraping the exposed path with Prometheus
Bonus: Scraping Envoy metrics from sidecar proxies

Let's start with this example of a minimal Consul Connect service defined in a Nomad job spec.

group "my-service" {
  network {
    mode = "bridge"
  }
  service {
    name = "my-service"
    port = 8080
    connect {
      sidecar_service {}
    }
  }
}

With this configuration, our service task is expected to listen for plaintext HTTP traffic on port 8080. A sidecar task will be created for us that runs Envoy, and it will be listening for HTTPS traffic on a dynamically assigned port. This traffic will be proxied to our service task.

This works great as long as we only ever need to talk to this service through a Connect proxy, but what if we're using Prometheus to scrape metrics from the service? For that, we'd prefer to use plain HTTP, without having to deal with the mutual TLS that the Connect proxy requires, and we need to be able to address each individual instance of the service, rather than have metrics scraping requests be load-balanced between instances.

Consul Connect supports doing this, but it's not well-documented how to set it up end-to-end all the way to Prometheus. It's not that hard once you know what to do, so let's see how.

Exposing the metrics outside your task container

Consul Connect's Envoy proxy supports designating specific URL paths that you want to expose as plain HTTP (no TLS) endpoints. Metrics and healthchecks are the main use cases this is meant for. When you expose a path, you designate a new Envoy listener port, separate from the one used for mutual TLS. HTTP requests to this port will proxy to your service, but only for the paths you designate. Other paths will get 404 responses.

There's a shortcut for doing this for Consul checks by setting expose = true on the check, but for other paths, you have to be more explicit by setting the expose configurations on your sidecar proxy:

group "my-service" {
  network {
    mode = "bridge"
    port "expose" {}
  }
  service {
    name = "my-service"
    port = 8080
    connect {
      sidecar_service {
        proxy {
          expose {
            path            = "/metrics"
            protocol        = "http"
            local_path_port = 8080
            listener_port   = "expose"
          }
        }
      }
    }
  }
}

We made two changes here. First, we added a new dynamic port to the group's network config named "expose". The name isn't important, it's just a label, but it does need to match the label used as the listener_port in the expose config. The "expose" port will give us an arbitrary available port on the host, and it will also be mapped to a port inside the group's network.

Now we've created the port, we define an expose path config for our proxy. Our service provides Prometheus metrics on the /metrics path, so that's what we want to expose. The local_bind_port should usually be the same as the service port: it's the port where we are already serving the path we want to expose. For listener_port, rather than use a hard-coded port number, we can use a label instead, which lets us have Nomad choose the port dynamically for each instance.

If you submit this job, you should be able to see /metrics under "Exposed Paths" for each service instance.

List of exposed paths for a service instance in Consul.

And in Nomad, for each allocation, you can see the "expose" port you declared.

The expose port entry on the allocation page in Nomad. Shows the IP plus port address for that port.

You may notice that the port you see in Consul and Nomad don't necessarily match. Nomad is showing the port that was assigned on the host: the one that is accessible from other machines. Consul is showing the port that was mapped to that the Envoy proxy is listening on inside the task group's network. The first one is the more interesting one for you: you can use it to test that your metrics are actually being exposed.

$ curl -i http://10.0.0.4:30801/metrics
HTTP/1.1 200 OK
content-type: text/plain; version=0.0.4; charset=utf-8
date: Sun, 21 Feb 2021 18:26:25 GMT
x-envoy-upstream-service-time: 1
server: envoy
transfer-encoding: chunked
# HELP http_server_duration
# TYPE http_server_duration histogram
http_server_duration_bucket{http_flavor="1.1",http_host="10.0.0.4:26994",http_scheme="http",http_server_name="Server",le="0.01"} 0
http_server_duration_bucket{http_flavor="1.1",http_host="10.0.0.4:26994",http_scheme="http",http_server_name="Server",le="0.025"} 0
http_server_duration_bucket{http_flavor="1.1",http_host="10.0.0.4:26994",http_scheme="http",http_server_name="Server",le="0.05"} 0
...

And just to make sure, let's see what happens if we try to access other paths besides metrics.

❯ curl -i http://10.0.0.4:30801/
HTTP/1.1 404 Not Found
date: Sun, 21 Feb 2021 18:28:08 GMT
server: envoy
content-length: 0

Perfect, so only our metrics are being exposed insecurely; the rest of the service is protected by the proxy.

Scraping the exposed path with Prometheus

Now we have a port where our metrics are exposed in a way that Prometheus could scrape them, but it has to know where to look. We can use consul_sd_configs on a scrape job to find all of our service instances, but unfortunately, the host port number for the expose port isn't available in any of the labels that gives us. In fact, right now, Consul doesn't know about that port number at all, so we need to tell it what that port is by adding some metadata to our service:

group "my-service" {
  network {
    mode = "bridge"
    port "expose" {}
  }
  service {
    name = "my-service"
    port = 8080
    
    meta {
      metrics_port = "${NOMAD_HOST_PORT_expose}"
    }
    connect {
      sidecar_service {
        proxy {
          expose {
            path            = "/metrics"
            protocol        = "http"
            local_path_port = 8080
            listener_port   = "expose"
          }
        }
      }
    }
  }
}

The trick here is that Nomad provides interpolation variables for each port you define in your network, and it supports using those variables in all kinds of interesting places, including the meta field of a service. When an instance of our service is registered, Nomad will substitute this with the port it assigned on the host side, and that data will now be available in Consul.

Metadata table for service instance in Consul, showing the metrics_port key with the port number we wanted

All values in meta are available to Prometheus when using Consul service discovery, so we can use this with some clever relabeling to target our scraping at the correct port.

- job_name: consul-services
  consul_sd_configs:
    - {}
  relabel_configs:
    # Don't scrape the extra -sidecar-proxy services that Consul Connect
    # sets up, otherwise we'll have duplicates.
    - source_labels: [__meta_consul_service]
      action: drop
      regex: (.+)-sidecar-proxy
    # Only scrape services that have a metrics_port meta field.
    # This is optional, you can use other criteria to decide what
    # to scrape.
    - source_labels: [__meta_consul_service_metadata_metrics_port]
      action: keep
      regex: (.+)
    # Replace the port in the address with the one from the metrics_port
    # meta field.
    - source_labels: [__address__, __meta_consul_service_metadata_metrics_port]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: ${1}:${2}
      target_label: __address__

And there it is! Prometheus should be able to find your services now and scrape their metrics endpoints, while other paths are protected with mutual TLS thanks to Consul Connect.

You can combine the relabeling configs shown above with other rules as you see fit. For instance, I prefer to use an explicit metrics_path meta field to decide what gets scraped, so I can use the same scrape job for both Connect and non-Connect services, potentially with non-standard metrics paths. That looks like this:

- job_name: consul-services
  consul_sd_configs:
    - {}
  relabel_configs:
    - source_labels: [__meta_consul_service]
      action: drop
      regex: (.+)-sidecar-proxy
    - source_labels: [__meta_consul_service_metadata_metrics_path]
      action: keep
      regex: (.+)
    - source_labels: [__meta_consul_service_metadata_metrics_path]
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_consul_service_metadata_metrics_port]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: ${1}:${2}
      target_label: __address__

Bonus: Scraping Envoy metrics from sidecar proxies

In addition to the metrics your service provides, the Envoy proxy that Connect uses also exposes many metrics that you may find useful and want to scrape. The process for doing so is similar, though not identical, to what we did before to expose our service metrics.

First, we need to configure the proxy to set up a listener for its metrics on a particular port, and then map a host port to that inside our task group.

group "my-service" {
  network {
    mode = "bridge"
    port "expose" {}
    port "envoy_metrics" {
      to = 9102
    }
  }
  service {
    name = "my-service"
    port = 8080
    
    meta {
      metrics_port       = "${NOMAD_HOST_PORT_expose}"
      envoy_metrics_port = "${NOMAD_HOST_PORT_envoy_metrics}"
    }
    connect {
      sidecar_service {
        proxy {
          expose {
            path            = "/metrics"
            protocol        = "http"
            local_path_port = 8080
            listener_port   = "expose"
          }
          config {
            envoy_prometheus_bind_addr = "0.0.0.0:9102"
          }
        }
      }
    }
  }
}

This time, while our port is assigned dynamically on the host side, we map it to a specific port number (9102) inside the task group. The envoy_prometheus_bind_addr config doesn't support interpolation, so we need to choose a specific port for it. That's okay, because that port number is local to our task group and won't conflict with other groups or jobs.

Once again, to make this port number available to Prometheus, we make sure the host port number assigned to it is set on the meta of the service instance. Now we can create another scrape job in Prometheus to get these metrics:

- job_name: consul-connect-envoy
  consul_sd_configs:
    - {}
  relabel_configs:
    - source_labels: [__meta_consul_service]
      action: drop
      regex: (.+)-sidecar-proxy
    - source_labels: [__meta_consul_service_metadata_envoy_metrics_port]
      action: keep
      regex: (.+)
    - source_labels: [__address__, __meta_consul_service_metadata_envoy_metrics_port]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: ${1}:${2}
      target_label: __address__

This looks very similar to the job for the service metrics, just with a different meta key.