Prometheus
Synopsis
Director scrapes one Prometheus metric endpoint, as the Prometheus server does: a node_exporter, an application's /metrics, kube-state-metrics, a federation endpoint, or any endpoint that speaks the Prometheus text exposition format. Nothing is installed beside the endpoint: Director asks it over HTTP or HTTPS, reads the answer as it streams, and writes what it found as ECS records.
One device, one endpoint. Scraping forty exporters takes forty devices. Finding endpoints (Kubernetes pods and services, DNS, Consul) is not part of the device: a device names one URL.
What to write is declared by datasets. A dataset is one scrape of the endpoint on its own interval, with the metrics it writes (patterns of metric names) and the labels it adds. Every sample a dataset wants is one record, and so is every scrape. A device runs any number of datasets: for example, the pods' state every 30 seconds and everything else every five minutes are two datasets of one device. Datasets that fall due together share one scrape.
The device dials the endpoint and binds no port, so any number of Prometheus devices can name port 9100, and none of them blocks a listener. One Director owns each device: two would scrape the endpoint twice and write every sample twice.
The Prometheus device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows Prometheus devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.
Schema
- id: <numeric>
name: <string>
description: <string>
type: prometheus
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
url: <string>
username: <string>
password: <string>
bearer_token: <string>
headers: <object>
timeout: <numeric|string>
max_response_size: <numeric|string>
tls:
status: <boolean>
server_name: <string>
ca_name: <string>
cert_name: <string>
key_name: <string>
passphrase: <string>
insecure_skip_verify: <boolean>
min_tls_version: <string>
max_tls_version: <string>
definitions: <definition[]>
Configuration
The following fields are used to define the device:
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | - | Unique numeric identifier |
name | Y | - | Device name |
description | N | - | Optional description |
type | Y | - | Must be prometheus |
tags | N | - | Optional tags |
pipelines | N | - | Optional pre-processor pipelines |
status | N | true | Enable/disable the device |
Endpoint
| Field | Required | Default | Description |
|---|---|---|---|
url | Y | - | http:// or https://, with a host and the path of the metrics. A wildcard host (0.0.0.0) is refused, and so is a user name or a password written into the URL: username, password, and bearer_token carry them. A query is kept (a federation endpoint's match[]); a query parameter shaped like a credential (api_key, token, and so on) is masked wherever the URL is shown, in logs and in records. |
timeout | N | 10 | Seconds the endpoint is given to answer whole (the connect, the handshake, the request, and the body read to its end), from 1 to 300. The time Director takes to write the samples it was handed does not count; see Limits. A duration of whole seconds (30s, 2m) is also accepted. The value is sent to the endpoint too, as X-Prometheus-Scrape-Timeout-Seconds. |
max_response_size | N | 64MB | How much of an answer is read, counted decompressed: a number of bytes, or a size such as 64MB or 1GiB (binary units), from 1 KiB to 1 GiB |
Authentication
| Field | Required | Default | Description |
|---|---|---|---|
username | N | - | Basic authentication user name, sent with password |
password | N | - | Basic authentication password. A password with no user name is refused. |
bearer_token | N | - | Sent as Authorization: Bearer <token>. It is one authentication: refused beside username. |
headers | N | - | An object of header names and values sent with every scrape. See below for the headers that are refused. |
Each of these takes plain text, ${ENV_VAR}, or $secret{...}, and is resolved at every scrape: a password or a token rotated in the vault is sent from the next scrape on, and the device does not restart. Credentials that cannot be read (a variable that is not set, a secret store out of reach) fail the scrape with the state Authentication error, and the message names the property, never its value.
In headers, Host is refused because it follows the URL, and Authorization is refused while a username or a bearer_token is set. Accept and Accept-Encoding are refused as well: the scrape sets the format and the compression it reads.
Credentials over an http:// URL are sent in cleartext. The device says so in its log when it starts, and runs.
A configuration error, such as a missing or malformed URL, a value out of its bounds, or a value that is not a number or a switch, stops the device at start with the state Configuration error, and the message names the property. A number is read whole: a fraction, or a number no 32-bit integer holds, is refused and never converted, and so is a ${ENV_VAR} that is not set. A switch is true or false, or text that says so; anything else is refused, never read as false.
TLS
The URL's scheme decides whether the scrape uses TLS: https:// does, http:// does not. On an http:// URL, the tls settings are not used.
| Field | Required | Default | Description |
|---|---|---|---|
tls.status | N | the scheme's | Refused when it contradicts the URL's scheme |
tls.server_name | N | the URL's host | The name the endpoint's certificate is checked against |
tls.ca_name | N | the system trust store | The authority of the endpoint's certificate |
tls.cert_name, tls.key_name | N | - | A client certificate and its key, for an endpoint that identifies its clients by one. Set both or neither. |
tls.passphrase | N | - | Passphrase for an encrypted key |
tls.insecure_skip_verify | N | false | Skip the check of the endpoint's certificate |
tls.min_tls_version, tls.max_tls_version | N | Go's defaults | tls1.0, tls1.1, tls1.2, or tls1.3; any other value is refused |
TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:
- File name — resolved relative to the service root directory. Nested paths such as
certs/prod/server.pemare supported. - Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
- Inline PEM content — used verbatim when the value contains
-----BEGIN. - Environment variable —
${ENV_VAR}. - Vault reference —
$secret{id=...}or$secret{store=...,ref=...}.
TLS material is read when the device starts, and material that cannot be read then stops it with Configuration error. Director reads it again whenever it compares the device's configuration: at every configuration change, and every few seconds for a device that names a $secret{...}. Material that changed where it is kept (a CA or a certificate rotated in its file, variable, or secret) restarts the device, which uses the new material from then on. Material that cannot be read at such a check, with the settings unchanged, restarts nothing: the device goes on with what it read, and logs this every ten minutes for as long as it lasts.
Scraping
The Request
A scrape is a GET of the URL. It asks for the text exposition format first (Accept: text/plain;version=0.0.4;q=1,*/*;q=0.1), and sends the scrape's timeout in X-Prometheus-Scrape-Timeout-Seconds and Director's User-Agent.
- Compressed. The answer is asked for compressed and decompressed as it is read.
- Kept-alive connections. Connections are kept between scrapes, so a scrape every few seconds does not open a connection and make a TLS handshake each time.
- Proxy. A Director whose environment names a proxy (
HTTPS_PROXY,HTTP_PROXY,NO_PROXY) scrapes through it.
Formats
A success (2xx) in the text exposition format, or with no content type given, is read as it streams, a sample at a time, and never held whole.
An answer in another format fails the scrape, and the error names the format: an HTML page or a JSON document (most often a URL that names a page rather than the endpoint's metrics), OpenMetrics, or the protobuf format. An endpoint that speaks those formats answers in the text format when asked for it first, as the scrape asks. An answer in an encoding the scrape did not ask for (deflate, br) also fails it.
A line the format cannot read, such as a malformed sample or a line longer than 1 MiB, is skipped and counted in prometheus.request.skipped; the rest of the answer is read.
Limits
timeout is the endpoint's time. It runs while the endpoint is connected to, asked, and read, and stands still while Director writes the samples it was handed through the dataset's pipeline to its queue. A pipeline or a queue that is slow for a moment does not fail a scrape that the endpoint answered in time. An endpoint that runs out of time fails the scrape, and the samples read before are kept.
An answer larger than max_response_size fails the scrape. The samples on the whole lines before the limit are written, and the line the limit cuts is not, since a value cut short is a value the endpoint never exposed. A body that breaks off is read the same way.
Redirects
Redirects are followed, at most 10. The credentials go to the URL's own origin only: a redirect to another scheme, host, or port is followed without the basic authentication, the bearer token, or the configured headers. No redirect carries a Referer, which would carry the URL's query to the origin redirected to.
Datasets
The device's definitions are its datasets. In the console they are the datasets and profiles assigned to the device in its prometheus_metrics_collector, the device's one dataset type:
| Dataset | Definition | Definition ID | Kind |
|---|---|---|---|
metrics | prometheus_metrics_collector | 17150 | metric |
- Any number of datasets. Each dataset is one scrape on its own interval, so a device can run several; datasets that fall due together share one scrape, and each writes what it wants of it.
- Pipelines. A dataset's records enter the pre-processing pipeline the dataset selects, or the device's when it selects none.
- Nothing assigned is allowed. The device then checks every minute that its endpoint answers, and writes nothing; its connection state follows the endpoint.
- A dataset that cannot be read is left out, for example one with an interval below 1, an invalid pattern, or an invalid label. A warning names it (a console dataset by its ID, one in a configuration file by its position), and the rest of the device runs. When every dataset assigned is left out, the device stops with Configuration error.
Dataset Properties
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | 60 | Seconds between the dataset's scrapes, 1 or more, in slots counted from its first scrape. See When Things Are Scraped. |
metric_include | N | all | Metric name patterns, a list or a single one: * for any run of characters, ? for one character, [...] for a set. A sample is written when its name matches one of them, or when there are none. At most 256. |
metric_exclude | N | - | The same form: a sample whose name matches one is not written, whatever metric_include says. At most 256. |
labels | N | - | Written on every record of the dataset as ECS labels. At most 32; a name is letters, digits, underscores, and hyphens (at most 64 characters, not starting with a digit or a hyphen), and a value is text, a number, or a switch of at most 256 characters. |
ignore_timestamp | N | false | When on, every sample is dated by the scrape, including samples the endpoint dates itself |
request_record | N | true | Write a record of every scrape; see Records |
pipeline_name | N | the device's | The pre-processing pipeline the dataset's records enter |
The patterns match the sample's name as the endpoint writes it. A histogram's _bucket, _sum, and _count samples, and a summary's _sum and _count samples, are matched by their own names: http_request_duration_seconds_* matches all of them.
When Things Are Scraped
- At start the device scrapes once for every dataset.
- Each dataset on its own interval after that, in slots counted from its first scrape: every 30 seconds from the start is 30, 60, and 90 seconds in, however late a scrape starts or ends. A dataset every 30 seconds and one every 300 share the scrape at every five minutes.
- A scrape longer than its interval skips the slots it ran over, as the Prometheus server skips them. The next scrape is the dataset's first slot after it ended, and none is made to catch up.
- Timestamps. A sample the endpoint does not date is dated when the answer came, all the samples of one scrape at one time, as the Prometheus server dates them. A sample the endpoint dates is dated by it, unless its dataset sets
ignore_timestamp. - On a stop the device ends its scrape at once, mid-answer: what was written is kept, and no record of the cut-short scrape is written. The device then flushes the queue it wrote to, and only then is it stopped, so a restarted device never has its predecessor still writing beside it. A flush that takes longer than 15 seconds is logged as a warning, and the device waits for it all the same.
Records
Every record is one ECS 8.11 document. These fields are on every record:
| Field | Value |
|---|---|
@timestamp | The sample's time (see When Things Are Scraped), or the scrape's start |
ecs.version | 8.11.0 |
event.module | prometheus |
event.dataset | prometheus.metric for a sample, prometheus.request for a scrape |
event.kind | metric |
event.category, event.type | host, info |
observer.name | The device's name in the console |
observer.hostname | The Director's host name |
observer.vendor, observer.product | VirtualMetric, Prometheus |
url.full | The URL, with a credential in its query masked |
url.scheme, url.domain, url.port, url.path | The URL's parts; the port is the scheme's when the URL names none |
server.address, server.port | The endpoint's host and port |
server.ip | The host, when it is an IP address |
network.protocol, network.transport | http, tcp |
labels.<name> | The dataset's labels, when it has any |
Samples
One prometheus.metric record is written for every sample the dataset wants:
| Field | Value |
|---|---|
prometheus.metric.name | The name as the endpoint writes it, suffix included (_total, _bucket, _sum, _count) |
prometheus.metric.type | counter, gauge, histogram, summary, or untyped, from the family's # TYPE line |
prometheus.metric.value | The value |
prometheus.labels.<name> | The labels the endpoint gave the sample, a bucket's le and a quantile's quantile among them |
A histogram's buckets and a summary's quantiles are samples of their own. A sample whose value is NaN or an infinity is not written, since JSON holds neither, and is counted in prometheus.request.non_finite. A summary with no observation yet, for example, gives NaN quantiles.
Scrapes
One prometheus.request record is written for every scrape, unless the dataset turns request_record off:
| Field | Value |
|---|---|
event.start, event.duration | When the scrape began, and how long it took with the answer read to its end (nanoseconds) |
event.outcome | success or failure |
error.type, error.message | For a failure: connection, dns, timeout, certificate, authentication, credentials, status, format, too_large, or redirect, and what happened |
http.response.status_code | The answer's status, when there was one |
http.response.body.bytes | How much of the body was read, decompressed |
http.response.mime_type | The answer's content type |
prometheus.request.samples | The samples the dataset wrote; a sample whose write failed is not counted |
prometheus.request.filtered | The samples its patterns left out |
prometheus.request.non_finite | The samples left out as NaN or an infinity |
prometheus.request.skipped | The lines the format could not read |
prometheus.request.lines | The lines of the answer, of every kind |
Device State
The connection state follows the latest scrape:
| The scrape | State |
|---|---|
| Read | Connected |
| No connection, a name that does not resolve, no answer in time, or too many redirects | Connection error |
| A certificate that does not verify | Certificate error |
401, 403, or 407, or credentials that cannot be read | Authentication error |
Another status, another format, or an answer past max_response_size | Error |
| A configuration that cannot run | Configuration error |
A failure is logged once for as long as it lasts (a repeat is held back for half an hour), and its end is logged. Its message says the same at every scrape that fails the same way: details that change from one connection to the next, such as the connection's own port, are left out of it. Records that could not be written are logged the same way and do not change the state, since the endpoint answered.
Configuration Changes
Any change to the device, or to a dataset assigned to it, restarts the device: its endpoint, how it asks it, its bounds, and its datasets. So does an edit that no longer validates, which the restart then reports as Configuration error. Credentials are compared as they are configured, which for a secret is its name and not its value: a secret rotated in its store is sent from the next scrape on and restarts nothing. TLS material is compared as it reads; see TLS.
The Prometheus device is not simulated. Its records are an endpoint's own metrics, a shape a generated sample cannot stand in for.
Examples
node_exporter
Scraping a database server's node_exporter every 30 seconds, leaving out the Go runtime and HTTP handler metrics... | |
Application Behind TLS with a Bearer Token
Scraping an application's metrics over TLS with a private CA, sending a bearer token read from an environment variable at every scrape... | |
kube-state-metrics on Two Intervals
Scraping kube-state-metrics behind a proxy that asks for a user and a tenant header: the pods' state every 30 seconds, the rest every five minutes without a record of each scrape... | |
Reachability Only
With no dataset assigned, the device only checks every minute that the endpoint answers, and writes nothing... | |