libvirt (KVM/QEMU)

Pull

Synopsis

Director polls one KVM/QEMU host through its libvirt daemon, over SSH, TLS, or TCP, or on the Director's own host, and writes what it reads as ECS documents: the host itself, its domains (guests) as they are configured, the host's own CPU and memory, and each domain's state, CPU, balloon, vCPU, network interface, and disk statistics. Nothing is installed on the host, and nothing of libvirt on the Director, which speaks libvirt's protocol itself.

It covers a KVM host managed directly through libvirt — with virsh, virt-manager, Cockpit, or an appliance built on them — which has no management engine above it for the oVirt / Red Hat Virtualization or Proxmox VE devices to read. It reads what Telegraf's libvirt input reads, and more: the host itself, the domains' configuration, and the host's own CPU and memory.

What the device reads is decided by the datasets and profiles assigned to it, as for the oVirt and Proxmox VE devices. Each data model is a dataset of its own — see libvirt Datasets.

The host is the device. One device reads every domain its daemon knows, running or not.

Nothing it finds is registered as a device. The host is the host.name on every record, and a domain is libvirt.domain and one of related.hosts on its records. None of them becomes a device of its own.

note

The libvirt (KVM/QEMU) device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows libvirt (KVM/QEMU) devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.

Use Cases

• An inventory of a KVM host that has no engine above it. The host, its hypervisor and libvirt versions, and every domain with its vCPUs, memory, disks, and network interfaces, with each change marked.
• How each guest is doing, with no agent on the host. Each domain's state, CPU time, and balloon, and its vCPUs', network interfaces', and disks' figures, every minute.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: libvirt
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    uri: <string>
    username: <string>
    private_key: <string>
    passphrase: <string>
    password: <string>
    host_key: <string>
    timeout: <numeric>
    tls:
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      server_name: <string>
      insecure_skip_verify: <boolean>
      min_tls_version: <string>
      max_tls_version: <string>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be libvirt
tags	N	-	Optional tags
pipelines	N	-	Optional pre-processor pipelines
status	N	true	Enable/disable the device

Connection

Field	Required	Default	Description
uri	Y	-	The libvirt connection URI, as virsh -c takes it. See Transports.
timeout	N	30	Seconds the connection and each call may take, from 1 to 300

Transports

The URI's transport decides how the device reaches the daemon, and which of the settings below it reads:

URI	Port	What the device does
qemu+ssh://user@host/system	22	Logs in over SSH and opens the daemon's socket on the host through the host's SSH server. See SSH.
qemu+tls://host/system, qemu://host/system	16514	TLS: verifies the daemon's certificate and shows it a client certificate. See TLS.
qemu+tcp://host/system	16509	Cleartext TCP, to a daemon that asks for no authentication (auth_tcp = "none")
qemu:///system	-	The daemon on the Director's own host

A port written in the URI, qemu+ssh://user@host:2222/system, wins over the default.

The read-only socket. For the SSH and the local transports, the socket parameter names the daemon's socket on its host, /var/run/libvirt/libvirt-sock by default. ?socket=/var/run/libvirt/libvirt-sock-ro reads through the read-only socket, which any local user may open and which serves every call the device makes:

qemu+ssh://monitor@kvm1.example.com/system?socket=/var/run/libvirt/libvirt-sock-ro

Refused in the URI, each with what to use instead:

• A password: use password.
• A user in a TLS or TCP URI: a TLS daemon checks the client certificate, and a user is read for qemu+ssh only.
• The ext transport.
• The parameters that name a file or a choice on the Director: keyfile (use private_key), known_hosts and known_hosts_verify (use host_key, or trust on first use), sshauth (use private_key or password), pkipath (use tls.ca_name, tls.cert_name, and tls.key_name), no_verify, and tls_priority. Any other parameter but socket is refused too.

An error never quotes the URI, which could carry a password.

warning

Over qemu+tcp, what the device reads crosses the network in cleartext, and the daemon has to take connections with no authentication, so anyone who reaches its port can manage the host's domains. Reach a host over a network with qemu+ssh or qemu+tls.

SASL is not spoken. A daemon that asks for SASL — auth_tcp = "sasl", libvirtd's default for TCP, auth_tls = "sasl", or auth_unix_rw = "sasl" — refuses the device, which reports it as Authentication error. The state says to reach the host over qemu+tls or qemu+ssh. Reach such a host over SSH, or over TLS with a client certificate where auth_tls does not ask for SASL.

SSH

For qemu+ssh only. Settings the URI's transport does not use are said once, as a warning, when the device starts.

Field	Required	Default	Description
username	N	the URI's user	The SSH user, when the URI names none. A user in the URI wins over it.
private_key	N	-	The SSH private key: its PEM, or a file under the Director's root
passphrase	N	-	The private key's passphrase
password	N	-	The SSH password, offered after the key
host_key	N	trust on first use	The host's SSH key: its SHA256 fingerprint, or the key itself as its .pub file or a known_hosts line holds it

A private_key or a password is required.

• The user must be able to open the socket: root, a member of the libvirt group (where polkit or the socket's mode lets the group in), or any user through the read-only socket. A user outside the libvirt group may too, when a polkit rule gives it org.libvirt.unix.manage; others are an Authentication error.
• The host's SSH server must forward the socket, which OpenSSH does unless AllowStreamLocalForwarding (or DisableForwarding) says no.
• The key is offered first, then the password.

The host's key is checked before any credential is sent. With host_key set, the key the host presents must match it. Without it, the key the host presents first is trusted and kept, per device, host, and port, in the Director cluster's replicated store, and required from then on. A host that presents another key is refused with Authentication error, and the error names every key the host presented, ready to pin.

Any of the host's keys may be pinned. The Director asks for ECDSA, then RSA, then Ed25519; when the host presents another key than the trusted one, it logs in again for each other type, once each, before any credential is sent. So a fingerprint of any key works, and a key presented in a certificate matches a pin of the key:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

TLS

For qemu+tls and qemu://host/system only.

Field	Required	Default	Description
tls.ca_name	N	the system trust store	The CA that signed the daemon's certificate, libvirt's cacert.pem
tls.cert_name	N	-	The client certificate the Director shows the daemon, libvirt's clientcert.pem
tls.key_name	N	-	The client certificate's private key, libvirt's clientkey.pem
tls.passphrase	N	-	The private key's passphrase, for an encrypted key
tls.server_name	N	the URI's host	The name the daemon's certificate is verified for
tls.insecure_skip_verify	N	false	true accepts any certificate. Set the CA instead where you can.
tls.min_tls_version	N	tls1.2	tls1.0, tls1.1, tls1.2, or tls1.3
tls.max_tls_version	N	-	No cap

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

The daemon checks the client certificate after the handshake, against its CA and its tls_allowed_dn_list, and closes the connection when it does not take it, which the device reports as Authentication error.

Credentials

Every credential — username, private_key, passphrase, password, and the tls material — takes plain text, ${ENV_VAR}, or $secret{...} ($secret{id=...} or $secret{store=...,ref=...}). They are resolved at every connection, so a secret rotated in the vault is used from the next connection on, and the device does not restart for it.

host_key is the exception: it is read as it is written, or from an environment variable, never from a vault.

Datasets

The device's definitions are its datasets, each named after its type, libvirt_<dataset>_collector. In the console they are the datasets and profiles assigned to the device. For the dataset types, their intervals, and their fields, see libvirt Datasets.

Connection States

State	When
Connected	The daemon answered and opened the URI, and at the next reading after a failure. A dataset the daemon refuses is named in the device's message.
Configuration error	A credential that does not resolve, a daemon with no driver for the URI, or a configuration that does not validate
Authentication error	The SSH login refused, an SSH host key that is not the pinned or trusted one, a client certificate the daemon did not take, polkit's refusal, or SASL asked for
Certificate error	The daemon's certificate does not verify
Initialization error	The Director's store of trusted SSH host keys did not answer. The login stops rather than trust whichever key answers.
Connection error	The daemon not reached, a connection lost, a call that outlived the timeout, or the daemon's socket the host's SSH server did not open for the user

A failure to connect at the start stops the device, which the Director starts again later.

Calls

• One connection per device, which its datasets share. Each call has the device's timeout as its deadline, and a call that outlives it closes the connection, which is the one way to end a call a daemon does not answer.
• A lost connection is dialed again once, by the next call: a daemon restarted, or a link the network dropped while it was idle. One lost during a call fails that reading, and the next reading dials again.
• The statistics do not wait for a busy domain. A domain in the middle of a migration, or whose monitor does not answer, has the figures that would wait left out, rather than holding every domain's back. A daemon before libvirt 4.5 does not support this; the device then asks without it for as long as the connection lasts.
• The host's capabilities are slow to read. libvirt builds them at every call, which can take more than a second; the host inventory reads them once a reading, hourly by default.

When Things Are Read

• Every dataset on its own interval, all of them at once when the device starts. The hourly inventories do not hold back the statistics, which are read every minute.
• A failed reading is read again after 60 seconds when its interval is longer. A reading the daemon refused — its access control denied a call, or its driver does not support it — waits for its interval, is said once in the device's log, and is named in the device's state until it reads again.
• What goes wrong in a dataset is its own. A refused call or a panic costs that dataset's reading alone, and the other datasets go on.

Records

Every record is one ECS 8.11 document:

• event.module is libvirt, and event.dataset is libvirt.<dataset>.
• event.kind is state for an inventory item, and metric for a statistics record.
• host.name is the host the daemon runs on.
• observer.vendor and observer.product are libvirt, observer.name is the device's name in the console, observer.version is the version of libvirt on the host, and observer.hostname or observer.ip is the URI's host, or the host's name for the local socket.
• A domain's records name it under libvirt.domain (name, uuid, id) and in related.hosts.

Inventory records carry event.action from a dataset's second reading on: added, changed, unchanged, or removed.

event.dataset: "libvirt.domain_stats" and libvirt.domain.name: "web01"
host.name: "kvm1"
observer.name: "kvm1"

Each record also carries its dataset's own definition id. For the fields of each dataset, see libvirt Datasets.

Console

In the console, go to Fleet Management > Devices > libvirt (KVM/QEMU) and click Add new device. The wizard's steps are:

1. General settings, as on every device.
2. Connection settings: URI and Timeout, then what the URI's transport reads:
   • For qemu+ssh, under SSH settings: Username (Optional), Password (Optional), Private key (Optional), and, once a private key is entered, Passphrase (Optional); then Host key (Optional).
   • For qemu+tls and qemu://host/system, under TLS settings: Skip certificate verification, the CA certificate, Server name (Optional), and the minimum and maximum TLS versions; then, under Client certificate, Certificate (Optional) and its Private key (Optional), with its passphrase.
   • For qemu+tcp, the wizard warns TCP is neither encrypted nor authenticated.
3. Data collection: the datasets or profile the device reads. A device receives datasets or a profile, never both.
4. Review and create.

The passwords, passphrases, and host key are masked where they are typed and on the device page, and the review shows only whether each is set. With no host key, the device page shows Trusted on first use.

Simulation

The libvirt (KVM/QEMU) device is not simulated. What it writes is decided by its datasets, seven record shapes under seven definition ids, and one generic sample cannot stand in for them. See Simulation.

Examples

SSH with a Key in a Vault

Reading a KVM host over SSH with a private key kept in a vault: the host and its domains hourly, and each domain's statistics every minute...

devices: - id: 512300170 name: kvm1 type: libvirt status: true properties: uri: "qemu+ssh://monitor@kvm1.example.com/system" private_key: "$secret{store=corp-vault,ref=libvirt/kvm1-monitor-key}" timeout: 30 definitions: - name: libvirt_host_collector - name: libvirt_domain_collector - name: libvirt_domain_stats_collector

Read-Only Socket with a Pinned Host Key

Reading through the read-only socket as an unprivileged user, with the host's ECDSA key pinned and the password in an environment variable...

devices: - id: 512300171 name: kvm2 type: libvirt status: true properties: uri: "qemu+ssh://kvm2.example.com/system?socket=/var/run/libvirt/libvirt-sock-ro" username: "monitor" password: "${KVM2_PASSWORD}" host_key: "SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8" definitions: - name: libvirt_host_stats_collector - name: libvirt_domain_stats_collector - name: libvirt_domain_disk_stats_collector

TLS with a Client Certificate

Reading a host over TLS, verifying the daemon against libvirt's CA and showing it a client certificate...

devices: - id: 512300172 name: kvm3 type: libvirt status: true properties: uri: "qemu+tls://kvm3.example.com/system" tls: ca_name: "libvirt/cacert.pem" cert_name: "libvirt/clientcert.pem" key_name: "$secret{store=corp-vault,ref=libvirt/kvm3-clientkey}" definitions: - name: libvirt_host_collector - name: libvirt_domain_collector - name: libvirt_domain_interface_stats_collector