Proxmox VE Datasets
Proxmox VE datasets define what a Proxmox VE device reads from a Proxmox VE cluster, or a node that stands alone, through its API (dataset type proxmox). They attach to devices of type proxmox, and in
Collector Types
There are six datasets, one per data model, each with its own definition id, event.dataset (proxmox.<dataset>), interval, and pipeline, so it can be searched, routed, and processed on its own. Four are inventories of what the cluster holds, each with how its items are doing; two are what happened on the cluster: the tasks its nodes ran and its cluster log. All six are read every 60 seconds by default, and no more often than every 10 seconds.
| Dataset (console title) | Dataset | Definition | Definition id | Kind | Default interval | Minimum |
|---|---|---|---|---|---|---|
| Proxmox VE Nodes | node | proxmox_node_collector | 25100 | inventory | 60 s | 10 s |
| Proxmox VE Virtual Machines | vm | proxmox_vm_collector | 25101 | inventory | 60 s | 10 s |
| Proxmox VE Containers | container | proxmox_container_collector | 25102 | inventory | 60 s | 10 s |
| Proxmox VE Storage | storage | proxmox_storage_collector | 25103 | inventory | 60 s | 10 s |
| Proxmox VE Tasks | task | proxmox_task_collector | 47 | log | 60 s | 10 s |
| Proxmox VE Cluster Log | log | proxmox_log_collector | 48 | log | 60 s | 10 s |
Every dataset takes these properties:
| Field | Required | Type | Default | Description |
|---|---|---|---|---|
interval | N | int | 60 | Seconds between readings. At least 10. |
pipeline_name | N | string | the device's | The preprocessing pipeline the dataset's records enter. |
initial_lookback | N | int | the device's | task and log only. Seconds before its first reading where the dataset starts when it has no checkpoint. 0 starts at the first reading. |
For the privileges the device's token needs for each dataset, see Token on the Proxmox VE device page.
Wizard fields
Every Proxmox VE dataset's
Rules
- Intervals. The inventories are read every minute, not hourly as other pollers' are, because their records carry how each node, guest, and storage is doing: an hourly reading would leave those figures an hour old. Their minimum is how often a node reports its figures to the cluster, every ten seconds: a shorter interval would only read again what the last reading wrote. An interval under the minimum is left out with a warning in the device's log, and the rest of the device runs.
- One of each. A dataset type is read once per device, however many datasets of it are assigned. The first decides its settings; the others are left out with a warning.
- Pipelines. A dataset's records enter the pipeline the dataset selects, otherwise the device's. Two datasets on one device can reach two pipelines.
- Requests. A reading of the guests or the storage is one request for the whole cluster. A reading of the nodes, and of the containers' swap, asks each online node once, at most four at a time.
Inventories
Inventory datasets write every item at every reading. From a dataset's second reading on, each record carries event.action:
event.action | Meaning |
|---|---|
added | Not in the last delivered reading |
changed | What it is differs from the last delivered reading's |
unchanged | As it was |
removed | Gone: written once, as it was last seen, dated at the reading |
- The baseline is the last delivered reading. A reading that fails, or whose records could not all be written, never becomes the one the next compares with, so no change goes unreported.
- What moves is not a change. An item is marked by what it is, never by its status or its use. A node's
status,uptime,load,ksm, and the use of its CPU, memory, swap, and root file system; a guest'sstatus,lock,ha,uptime,network, and the use of its CPU, memory, disk, and swap; and a storage'sstatusand space are written but not compared. A guest that migrated to another node changed, and so did one resized, retagged, or moved to another pool; a node upgraded, given another CPU or memory, or a new certificate changed. - Known in part is not removed. A node that is not online, or whose status did not answer, is written with what the cluster knows of it and not marked, and kept as the last reading that read it held it. So is a guest on a node that is not online, written with its status
unknownand none of its figures, and a container whose node did not say its swap. - A token that may audit nothing is refused the dataset, which keeps its last delivered reading: a revoked privilege does not mark every guest removed.
- The first reading after a start marks nothing: the baseline is kept in memory.
Fields sit under proxmox.<dataset>.
| Dataset | One record per | Fields |
|---|---|---|
node | Node of the cluster | name, id (node/pve1), status (online, offline, unknown), level (the subscription level), ssl_fingerprint, nodeid and ip (its corosync id and address; a node standing alone has no nodeid), version (its pve-manager version), kernel.name/release/version/machine, boot.mode/secure_boot, cpu.model/sockets/cores/threads/mhz/hvm/usage/iowait, memory, swap, and rootfs (total, used, free, available, used_pct), load.1/5/15, uptime (seconds), ksm.shared; host.ip, host.architecture, host.os.kernel. A node that is not online has its name, id, status, certificate fingerprint, corosync id, and address, and none of the rest. |
vm | Virtual machine (QEMU) | vmid, type (qemu), name, node, template, tags, pool, lock (backup, migrate, snapshot, …), ha.state, status (running, stopped, paused, …), uptime, cpu.count/usage, memory.total/used/free/used_pct/host, disk.total/read_bytes/write_bytes, network.in_bytes/out_bytes; related.hosts. A virtual machine's root disk use is not known to the cluster and is left out. Templates are written, with template: true. |
container | Container (LXC) | As vm, with type lxc, disk.used/free/used_pct, and swap.total/used/free/used_pct from its node |
storage | Storage of each node | id (storage/pve1/local-zfs), name, node, type (dir, lvmthin, zfspool, nfs, rbd, …), content (images, rootdir, backup, iso, vztmpl, snippets), shared, status (available, or unknown when its node did not report it lately), total, used, free, used_pct. A shared storage (NFS, Ceph, iSCSI) has a record on every node it is enabled on. |
Tasks
task reads every task each online node ran and ended since its checkpoint, from the node's own task archive, and writes each task once, ended, with its outcome, in the order they ended, dated when it ended. A task still running is written by the reading after it ends.
| Dataset | One record per | Fields |
|---|---|---|
task | Task a node ran and ended | proxmox.task.upid, node, type (the operation: qmstart, qmigrate, qmclone, qmdestroy, vzdump, vzcreate, …), id (the object it ran on: a guest's vmid for a guest's), user, token (when a token started it), pid, pstart, status (OK, WARNINGS: <n>, or the error it ended with), status_type (ok, warning, error, unknown), start, end, duration (seconds); event.action (the type), event.id (the UPID), event.outcome (success for OK or warnings, failure, or unknown), event.start, event.end, event.duration, user.name and user.domain (its user and realm), message (vzdump 102 on pve2 by backup@pve: WARNINGS: 1) |
- Node by node. Each node keeps its own archive of the tasks it ran, about a thousand, and the checkpoint holds, by node, the newest task a reading met there. A node that is not online is not asked, and the reading after it is back writes its tasks. A node no reading met yet, such as one that joined the cluster, is read from the dataset's start.
- An archive that rotated past the checkpoint is said in the device's log. The reading then takes the tasks that ended since five minutes before the newest one delivered, but those the checkpoint remembers; what the rotation took away is lost.
Cluster Log
log reads every entry of the cluster log since its checkpoint, dated when it was logged. The cluster keeps the log on every node in a ring of 128 KiB, merged from every node's entries, and a reading reads all of it.
| Dataset | One record per | Fields |
|---|---|---|
log | Entry of the cluster log | proxmox.log.uid and node (the node's number of the entry, and the node), id (<uid>:<node>), tag (the daemon: pvedaemon, pveproxy, pvescheduler, …), pid, user, priority (0–7), upid (when it names a task); message, log.level (err, info, …), log.syslog.severity, process.name, process.pid, user.name, event.id (<node>:<time>:<uid>) |
What the API itself writes to the log is named as ECS names it:
| Entry | event.action | event.category | Notes |
|---|---|---|---|
starting task UPID:... | task-started | process | proxmox.log.upid |
end task UPID:... <status> | task-ended | process | event.outcome from the status |
successful auth for user '<user>' | logged-in | authentication | user.name is who signed in; the entry is logged as root@pam's |
changed password for user '<user>' | password-changed | iam | user.target.name is whose password changed |
Sign-ins are in each node's syslog. Failed sign-ins are in no cluster log, and Proxmox VE drops some of its own writes to it (writing cluster log failed: ipcc_send_rec[7] failed: Invalid argument). Forward each node's syslog or journal to a Syslog device. A refused API token logs nothing. The task archive is the complete record of tasks.
The sign-in lines, as a Proxmox VE 8.4 node writes them to its syslog, with the user and the reason at their end:
pvedaemon[896]: authentication failure; rhost=::ffff:192.0.2.50 user=root@pam msg=Authentication failure
pvedaemon[895]: authentication failure; rhost=::ffff:192.0.2.50 user=monitor@pve msg=no password set
pvedaemon[896]: authentication failure; rhost=::ffff:192.0.2.50 user=ghost@pve msg=no such user ('ghost@pve')
pvedaemon[894]: <root@pam> successful auth for user 'alice@pve'
A user of the pam realm adds PAM's own line before the first (pam_unix(proxmox-ve-auth:auth): authentication failure; ...). rhost is the address the request came from, an IPv4 address written as ::ffff:, and a failure is written after the API's delay of about two seconds. Proxmox VE 8 logs to the journal, with no /var/log/syslog unless rsyslog is installed. A refused API token is answered 401 (invalid token value!), with no line here or in the cluster log.
The cluster keeps only its newest 128 KiB of log, a few hundred to a few thousand entries. An entry the ring drops between two readings is not read. A reading every minute meets every entry while the cluster logs less than the ring holds between two readings.
Checkpoints
The tasks and the cluster log resume from where the last delivered reading stopped:
- Across a restart or a failover. The checkpoint is kept, node by node, in the Director cluster's replicated store, per dataset, so a restart, or another node after a failover, resumes from it. A checkpoint that cannot be read fails the reading, which is tried again: a dataset never starts over while its checkpoint may say where it stopped.
- Written after delivery. A reading's checkpoint is stored only once its records are delivered. A crash before then writes those records again: at least once, never lost.
event.idis the key to recognize a repeat by. - The first reading of a dataset with no checkpoint writes the tasks or entries of the last
initial_lookbackseconds: the dataset's own, else the device's, else a day.
Examples
Reading the guests every 30 seconds into their own pipeline, the storage every five minutes, and the tasks and cluster log from an hour back... | |