Skip to main content

libvirt Datasets

libvirt datasets define what a libvirt (KVM/QEMU) device reads from a KVM/QEMU host through its libvirt daemon (dataset type libvirt). They attach to devices of type libvirt, and in Datasets & Profiles the category is labelled libvirt (KVM/QEMU). To create and assign datasets through the web interface, see Management. For the dataset and profile model, see Overview.

Collector Types​

There are seven datasets, one per data model, each with its own definition id, event.dataset (libvirt.<dataset>), interval, and pipeline, so it can be searched, routed, and processed on its own. Two are inventories of what the host holds, read hourly; five are how the host and its domains are doing, read every minute.

Dataset (console title)DatasetDefinitionDefinition idKindDefault intervalMinimum
libvirt Hosthostlibvirt_host_collector26100inventory3,600 s60 s
libvirt Domainsdomainlibvirt_domain_collector26101inventory3,600 s60 s
libvirt Host Statisticshost_statslibvirt_host_stats_collector26150metric60 s10 s
libvirt Domain Statisticsdomain_statslibvirt_domain_stats_collector26151metric60 s10 s
libvirt Domain vCPU Statisticsdomain_vcpu_statslibvirt_domain_vcpu_stats_collector26152metric60 s10 s
libvirt Domain Interface Statisticsdomain_interface_statslibvirt_domain_interface_stats_collector26153metric60 s10 s
libvirt Domain Disk Statisticsdomain_disk_statslibvirt_domain_disk_stats_collector26154metric60 s10 s

Every dataset takes these properties:

FieldRequiredTypeDefaultDescription
intervalNintper datasetSeconds between readings. At least the dataset's minimum above.
pipeline_nameNstringthe device'sThe preprocessing pipeline the dataset's records enter.

Wizard fields​

Every libvirt dataset's Dataset configuration step says how the dataset is read and has an Interval (Optional) field, in seconds: left empty, the dataset is read at its default; otherwise at least its minimum.

Rules​

  • Intervals. An interval under the dataset's minimum is left out with a warning in the device's log, and the rest of the device runs. The inventories are read hourly by default, as the oVirt, VMware, and Sangfor inventories are: what they hold is how the host and its domains are configured, and how each domain is doing is a statistics dataset of its own, read every minute.
  • One of each. A dataset type is read once per device, however many datasets of it are assigned. The first decides its settings.
  • Pipelines. A dataset's records enter the pipeline the dataset selects, otherwise the device's. Two datasets on one device can reach two pipelines.
  • Every domain. The device reads every domain the daemon lists to its user; there is no filter on which domains are read. The daemon's access control can narrow what it lists.

Inventories​

Inventory datasets write every item at every reading. From a dataset's second reading on, each record carries event.action:

event.actionevent.typeMeaning
addedcreationNot in the last delivered reading
changedchangeIts fields differ from the last delivered reading's
unchangedinfoAs it was
removeddeletionGone: written once more, as it was last seen
  • Keys. The host is keyed by its name, a domain by its UUID, so a renamed domain is changed.
  • The baseline is the last delivered reading. A reading that fails, or whose records could not all be written, never becomes the one the next compares with, so no change goes unreported.
  • Unread is not removed. A domain whose calls did not all answer is written with what was read and not marked, said once in the device's log, and compared at the next reading with the last reading that read it whole. A domain gone between the list and its own calls is removed.
  • What moves while a domain stays the same is written but not compared: its id (a new one at every start), its state and state_reason, memory.current (what its balloon holds), and each interface's target (named afresh at every start). The host's cpu.mhz, the speed its CPUs run at now, is not compared either.
  • A domain's configuration, not its runtime. A running domain's record is its live definition, but a running domain that is defined is compared by its persistent definition, as a shut-off one is, so what libvirt adds to a running domain's definition (a virtual network's bridge, a host-model CPU expanded into a model) does not mark it. A domain started, stopped, or migrated in is unchanged; one given another disk, interface, CPU, vCPU, or memory is changed, a shut-off one included.
  • The first reading after a start marks nothing: the baseline is kept in memory, so a restarted device, or one another Director takes over, marks nothing at its first reading.

Fields sit under libvirt.<dataset>.

DatasetOne record perFields
hostThe host the daemon runs onhost.hostname, host.architecture, host.id (the host's UUID); libvirt.host.name, uri (the daemon's own, qemu:///system), version (libvirt's), hypervisor.type/version (QEMU, 8.2.2), uuid, cpu.arch, cpu.model (libvirt's name for the model, Icelake-Server-noTSX), cpu.vendor, cpu.microcode, cpu.name (the firmware's brand string), cpu.sockets, cpu.cores (per socket), cpu.threads (logical CPUs online), cpu.threads_per_core, cpu.mhz, memory.total (bytes), numa.nodes, iommu, migration.live/transports, security_models (apparmor, selinux, dac), domain_types (kvm, qemu: whether KVM acceleration is there), guest_arches, and from the SMBIOS tables system.manufacturer/product/version/serial/uuid/sku/family and bios.vendor/version/date/release. Not compared: cpu.mhz.
domainDomain (guest) the daemon knows, running or notlibvirt.domain.name, uuid, id (while it runs), title, type (kvm, qemu), state and state_reason (as virsh domstate --reason names them: running/booted, paused/user, shutoff/destroyed), persistent, autostart, os.type/arch/machine/firmware/loader/secure_boot, cpu.mode/model, vcpu.current/maximum, memory.maximum/current (bytes, whatever unit the definition uses), graphics (vnc, spice), disks, and interfaces (below); related.hosts. Not compared: id, state, state_reason, memory.current, each interface's target.

Each of a domain's disks carries device (disk, cdrom, floppy, lun), type (file, block, network, volume, dir), target and bus (where the guest sees it), source (a file, a device, or a directory), pool and volume, or protocol, name, and hosts (a network disk: Ceph RBD, iSCSI, NBD, HTTPS), driver, format, cache, serial, readonly, and shareable. A network disk's name is written without its query, which can carry a token, and its authentication is never read.

Each of its interfaces carries type (network, bridge, direct, user, …), mac, network, portgroup, bridge, source_device and mode (a host device's), model, target (the device that stands for it on the host while the domain runs, vnet0), link_state, and mtu.

A running domain's definition is read without what only a read-write connection may see, such as a graphics password.

A daemon that cannot read the host's SMBIOS tables (no dmidecode, a driver that does not say, or a host that exposes none) leaves system, bios, and cpu.name out, which is said once in the device's log. What the firmware does not give, which libvirt passes on as dmidecode prints it, is left out too: Not Specified and <BAD INDEX> values are not written.

Statistics​

Each record of a domain's statistics names its domain under libvirt.domain (name, uuid, id) and in related.hosts; a host_stats record names the host alone, as host.name.

  • Counters are running totals: the host's since the host started, and a domain's since the domain started, as Telegraf writes them.
  • Sizes libvirt gives in KiB are written in bytes.
  • No zero for what the hypervisor does not give. A statistic the hypervisor does not give — a balloon with no guest driver, a vCPU's wait on an older kernel — is left out, never written as zero.
  • States are names, as virsh writes them, not numbers. A number no name is known for is written as its number.
DatasetOne record perFields
host_statsThe hostlibvirt.host_stats.cpu.kernel.ns, cpu.user.ns, cpu.idle.ns, cpu.iowait.ns (and cpu.intr.ns where the host gives it), memory.total.bytes, memory.free.bytes, memory.buffers.bytes, memory.cached.bytes, and memory.used.bytes and memory.used.pct as free(1) counts them
domain_statsDomain, running or notlibvirt.domain_stats.state, state_reason, cpu.time.ns, cpu.user.ns, cpu.system.ns, balloon.current/maximum/swap_in/swap_out/unused/available/usable/rss/disk_caches (each .bytes), balloon.major_faults/minor_faults/hugetlb_pgalloc/hugetlb_pgfail, balloon.last_update, vcpu.current, vcpu.maximum
domain_vcpu_statsvCPU of a running domainlibvirt.domain_vcpu_stats.id, state (running, blocked, offline), time.ns, wait.ns, delay.ns, halted (QEMU gives it on s390x, not x86)
domain_interface_statsNetwork interface of a running domainlibvirt.domain_interface_stats.name (vnet0), received.bytes/packets/errors/dropped, and the same under transmitted, from the domain's side
domain_disk_statsDisk the hypervisor gives a figure oflibvirt.domain_disk_stats.name (vda), path, backing_index, read.requests/bytes/time.ns, the same under write, flush.requests, flush.time.ns, errors, and allocation, capacity, physical, threshold (each .bytes; threshold is set by virsh domblkthreshold)

A domain that does not run has its state, its balloon, and its vCPU counts in domain_stats from its definition, and no CPU time: libvirt reads the figures of a domain with no process from its own process, so the CPU time it would give one is the daemon's. It has no domain_vcpu_stats or domain_interface_stats records. A disk whose image libvirt reads on the host (a file, a block device) has a domain_disk_stats record of its sizes alone (allocation, capacity, physical); a network disk has none.

Compared with Telegraf​

The datasets read what Telegraf's libvirt input reads, with these differences:

  • A dataset per model. Telegraf writes each statistics group as a measurement; here each group of a model of its own is a dataset.
  • Not read: Telegraf's perf, iothread, memory (bandwidth monitors), and dirtyrate groups, and its vcpu_mapping. They need host features (perf events, Intel RDT) or a calculation a user starts.
  • No domains filter. The device reads every domain the daemon lists to its user.
  • Names and bytes. The state and its reason are names, not numbers, and the balloon is in bytes, not KiB.
  • Read only here: the host, the domains' configuration, and the host's own CPU and memory.

Examples​

Reading the domains' disk statistics every 30 seconds into their own pipeline, each domain's statistics every minute, and the domains' configuration every 15 minutes...

devices:
  - id: 512300170
    name: kvm1
    type: libvirt
    status: true
    properties:
      uri: "qemu+ssh://monitor@kvm1.example.com/system"
      private_key: "$secret{store=corp-vault,ref=libvirt/kvm1-monitor-key}"
    definitions:
      - name: libvirt_domain_disk_stats_collector
        status: true
        properties:
          interval: 30
          pipeline_name: "kvm-disk-stats"
      - name: libvirt_domain_stats_collector
        status: true
      - name: libvirt_domain_collector
        status: true
        properties:
          interval: 900