Proxmox VE

Pull

Synopsis

Director polls one Proxmox VE cluster, or a node that stands alone, through its API, and writes what it reads as ECS documents: the cluster's nodes, its virtual machines (QEMU) and containers (LXC), each with how it is doing, its storage, the tasks its nodes ran, and its cluster log. Nothing is installed on the nodes.

What the device reads is decided by the datasets and profiles assigned to it, as for the oVirt, Sangfor HCI, and VMware devices. Each data model is a dataset of its own — see Proxmox VE Datasets.

The cluster is the device. Any node answers the API for the whole cluster, and hands what another node holds over to it, so one device reads every node, guest, and storage of the cluster through the node its URL names. Telegraf's proxmox input reads the guests of the one node it is told to; this device reads every node.

Nothing it finds is registered as a device. A node is the host.name on its own records and on those of the guests, storage, tasks, and log entries on it (see Tags). None of them becomes a device of its own.

One node answers for the cluster. The device does not move to another node when the one its URL names goes down. Name a node that is up, and put a load balancer in front of the cluster's nodes and name it when the device has to outlive one node.

The device dials the API and binds no port, so any number of Proxmox VE devices may use port 8006. One Director polls each device: two Directors never write the same task or log entry twice.

note

The Proxmox VE device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows Proxmox VE devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.

Use Cases

One inventory of a cluster. Its nodes, virtual machines, containers, and storage, and how each is doing, every minute.
An audit trail of the cluster. The tasks its nodes ran — who started, stopped, migrated, cloned, backed up, or deleted what — and the cluster log, sign-ins and password changes among it.

Token

The device signs in with a Proxmox VE API token, which needs no login, no ticket, and no CSRF token. It reads; it changes nothing.

PVEAuditor on / reads every dataset but the cluster log, which needs Sys.Syslog on / as well. PVEAuditor does not hold it.

A role of the five privileges the device reads with reads every dataset:

pveum role add VirtualMetric --privs "Sys.Audit,Sys.Syslog,VM.Audit,Datastore.Audit,Pool.Audit"
pveum user add monitor@pve
pveum acl modify / --roles VirtualMetric --users monitor@pve
pveum user token add monitor@pve virtualmetric --privsep 1
pveum acl modify / --roles VirtualMetric --tokens 'monitor@pve!virtualmetric'

A token with privilege separation (--privsep 1, the default) holds what both it and its user are given; one without holds its user's.

Dataset	Needs	Without it
Nodes	`Sys.Audit` on `/nodes/<node>`; on `/` for the addresses	A node's status is refused, and the dataset reads nothing. Without it on `/`, the nodes are written without their addresses and corosync ids, and not marked.
Virtual machines	`VM.Audit` on the guests, or on a pool of them	The API lists no guest the token may not audit. A token that may audit none is refused the dataset, rather than told every guest is gone.
Containers	`VM.Audit` on the guests, or on a pool of them	The same
Storage	`Datastore.Audit` on the storage, or on a pool of it	The same, for storage
Tasks	`Sys.Audit` on `/nodes/<node>`	A node the token may not audit is not read, and the reading says so. A token that may audit no node is refused the dataset.
Cluster log	`Sys.Syslog` on `/`	The dataset is refused.

A dataset the token may not read keeps the device Connected, since the API answered. The device's log says so once, with the privilege and the path, and again only once the dataset reads; the device's message names each such dataset, and why, until it reads:

Proxmox poller connected to pve-lab. proxmox_log_collector reads nothing: the token holds no Sys.Syslog on /, so the API answers it only the entries of the cluster log it wrote itself

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: proxmox
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    token: <string>
    token_id: <string>
    timeout: <numeric>
    initial_lookback: <numeric>
    tls:
      insecure_skip_verify: <boolean>
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `proxmox`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Connection

Field	Required	Default	Description
`url`	Y	-	The API of a node of the cluster, or of a load balancer in front of its nodes: `https://<host>:8006`. With no port it is `8006`, and with no scheme `https://`. `/api2/json` at its end is taken off, and a path before it is kept, for a proxy that serves the API under one. A user, a query, a fragment, or a wildcard host is refused.
`token_id`	N	-	The API token's id, `user@realm!tokenid` (`monitor@pve!virtualmetric`)
`token`	Y	-	The token's secret; or, with no `token_id`, the whole token, `USER@REALM!TOKENID=SECRET`, as Telegraf's `api_token` takes it

token_id and token each take plain text, ${ENV_VAR}, or $secret{...} ($secret{id=...} or $secret{store=...,ref=...}). They are resolved at every request, so a secret rotated in the vault is sent from the next request on. The configuration keeps the reference, and a changed reference restarts the device. A token id written as text is checked at start, and so is a secret (no space, no =); a reference is checked when it resolves.

TLS

Whether TLS is used is the URL's scheme: there is no tls.status.

Field	Required	Default	Description
`tls.insecure_skip_verify`	N	`false`	`true` accepts any certificate. Pin the CA instead where you can.
`tls.ca_name`	N	the system trust store	A CA to verify the API's certificate against
`tls.server_name`	N	the URL's host	The name the certificate is verified for, when it is not the URL's host
`tls.min_tls_version`	N	`tls1.2`	`tls1.0`, `tls1.1`, `tls1.2`, or `tls1.3`
`tls.max_tls_version`	N	-	No cap

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Pin the cluster's CA. Every node presents a certificate the cluster's own CA signed, unless another was uploaded. Set tls.ca_name to that CA, /etc/pve/pve-root-ca.pem — the same file on every node — rather than turning verification off.

warning

An http:// URL sends the API token in cleartext with every request, and anyone who reads it can call the API as its user until the token is deleted or expires. Use plain HTTP only for an endpoint behind a proxy that terminates TLS, where the network between the Director and the proxy is trusted. The device's log says so once at start.

Datasets

The device's definitions are its datasets, each named after its type, proxmox_<dataset>_collector. In the console they are the datasets and profiles assigned to the device. For the dataset types, their intervals, and their fields, see Proxmox VE Datasets.

Advanced Configuration

Field	Required	Default	Description
`timeout`	N	`60`	Seconds each request may take, from `1` to `300`
`initial_lookback`	N	`86400`	Seconds before their first reading where the tasks and the cluster log start when they have no checkpoint. `0` starts at the first reading. The dataset's own `initial_lookback` wins.

Connection States

A configuration error — no URL, no token, a malformed one, a reference that does not resolve, a CA that cannot be read — stops the device with Configuration error. The message names the property but never its value.
At start the device reads the API's version, which any token may read: the API answering, and taking the token, is what connected means. A token the API refuses (401: its secret is wrong, it expired or was deleted, its user is disabled) sets Authentication error and stops the device; the Director starts it again later.
An API that does not answer, or a gateway in front of it answering 502, 503, or 504, sets Connection error; one whose certificate does not verify, Certificate error.
A refused token is resolved again, and the request sent once more when it resolves to another one, such as a secret rotated in the vault since. The same token refused again fails the reading with Authentication error.
A node the API could not reach (595, 596) moves nothing: the node that answered is up.

The connection state follows the API: the device is Connected once it reads the API's version, and whenever a reading succeeds.

Requests

Deadlines. Every request has the device's timeout as its deadline, and stopping the device abandons the requests in flight.
Only the configured node is called. The device builds every request itself on the URL it was configured with, and never follows a redirect, so the token never leaves for another host. An answer larger than 64 MiB is not read.
At most four at once. A dataset that asks every node sends at most four requests at once.
User agent. The device identifies itself as VirtualMetric Director <version>.

When Things Are Read

Every dataset on its own interval, every minute by default, all of them at once when the device starts.
A failed reading is read again after 60 seconds when its interval is longer. A reading the token may not make waits for its interval, since it would be refused again.
What goes wrong in a dataset is its own. A refused request, records that cannot be written, or a panic costs that dataset's reading alone. The dataset keeps its checkpoint or its baseline, and the other datasets go on.
What one node cannot answer — a node that is not online, or one whose status, containers' swap, or task archive did not answer — is noted, and the reading goes on with the others. The device's log says what a dataset reads in part once, until it changes, and says so when it reads in full again.

Field	Holds
`observer.vendor`	`Proxmox`
`observer.product`	`Proxmox VE`
`observer.version`	The version of the node the URL names (`8.2.4`; `7.4-16` before Proxmox VE 8), read at start
`observer.name`	The device's name in the console: the cluster the record came through
`observer.hostname` or `observer.ip`	The URL's host
`host.name`	The node the record is about, or the node its guest, storage, task, or entry is on

Records

Every record is one ECS 8.11 document:

event.module is proxmox, and event.dataset is proxmox.<dataset>.
event.kind is state for a node, a guest, or a storage, and event for a task or a log entry.
Fields ECS has no name for sit under proxmox.<dataset>. Sizes are bytes, a CPU's usage and every used_pct are fractions of 1 (0.7847), times are UTC, and a figure the API leaves out, writes as null, or writes as no number is left out, never written as 0.

Inventory records carry event.action from a dataset's second reading on: added, changed, unchanged, or removed.

Each record also carries its dataset's own definition id. For the fields of each dataset, see Proxmox VE Datasets.

Console

In the console, go to Fleet Management > Devices > Proxmox VE and click Add new device. The wizard's steps are:

General settings, as on every device.
Connection settings: URL, a protocol and an Address. Under Authentication, Token ID (Optional) and Token secret; leave Token ID (Optional) empty to enter the whole token in Token secret. For HTTPS only, under TLS settings: Skip certificate verification, CA certificate (Optional), Server name (Optional), Minimum TLS version (Optional), and Maximum TLS version (Optional). With HTTP selected, the wizard warns that the token is sent in cleartext.
Advanced settings: Timeout and Initial lookback, both in seconds.
Data collection: the datasets or profile the device reads. A device receives datasets or a profile, never both.
Review and create.

The token secret is masked where it is typed and on the device page, and the review shows only whether it is set.

Simulation

The Proxmox VE device is not simulated. What it writes is decided by its datasets, six record shapes under six definition ids, and one generic sample cannot stand in for them. See Simulation.

Examples

Token in a Vault

Reading a cluster's nodes, guests, and storage every minute, with its tasks and cluster log, and the token's secret kept in a vault...

devices:
  - id: 512300160
    name: pve-lab
    type: proxmox
    status: true
    properties:
      url: "https://pve1.example.com:8006"
      token_id: "monitor@pve!virtualmetric"
      token: "$secret{store=corp-vault,ref=proxmox/monitor}"
    definitions:
      - name: proxmox_node_collector
      - name: proxmox_vm_collector
      - name: proxmox_container_collector
      - name: proxmox_storage_collector
      - name: proxmox_task_collector
      - name: proxmox_log_collector

Pinned Cluster CA

Verifying a node against the cluster's own CA, with the whole token in an environment variable, and starting the tasks an hour back...

devices:
  - id: 512300161
    name: pve-prod
    type: proxmox
    status: true
    properties:
      url: "https://pve2.example.com:8006"
      token: "${PVE_API_TOKEN}"
      initial_lookback: 3600
      tls:
        ca_name: "pve-root-ca.pem"
    definitions:
      - name: proxmox_vm_collector
      - name: proxmox_container_collector
      - name: proxmox_task_collector

Synopsis​

Use Cases​

Token​

Schema​

Configuration​

Device​

Connection​

TLS​

Datasets​

Advanced Configuration​

Connection States​

Requests​

When Things Are Read​

Tags​

Records​

Console​

Simulation​

Examples​

Token in a Vault​

Pinned Cluster CA​