VMware vSphere Datasets

VMware vSphere datasets define what a VMware vSphere device reads from a vCenter Server or a standalone ESXi host (dataset type vmware). They attach to devices of type vmware, and in Datasets & Profiles the category is labelled VMware vSphere. To create and assign datasets through the web interface, see Management. For the dataset and profile model, see Overview.

Collector Types

There are 41 datasets, one per data model, each with its own definition id, event.dataset, interval, and pipeline, so it can be searched, routed, and processed on its own. Each dataset's event.dataset is vmware.<dataset>.

Dataset (console title)	Dataset	Definition	Definition id	Kind	Default interval	Minimum
VMware ESXi Hosts	host	vmware_host_collector	6100	inventory	3,600 s	60 s
VMware ESXi Host Status	host_status	vmware_host_status_collector	6150	metric	60 s	10 s
VMware Events	event	vmware_event_collector	40	event log	60 s	10 s
VMware Tasks	task	vmware_task_collector	41	event log	60 s	10 s
VMware ESXi Host Processors	host_processor	vmware_host_processor_collector	6101	inventory	3,600 s	60 s
VMware ESXi Host Fans	host_fan	vmware_host_fan_collector	6102	inventory	3,600 s	60 s
VMware ESXi Host Power Supplies	host_power_supply	vmware_host_power_supply_collector	6103	inventory	3,600 s	60 s
VMware ESXi Host Sensors	host_sensor	vmware_host_sensor_collector	6104	inventory	3,600 s	60 s
VMware ESXi Host Storage Adapters	host_hba	vmware_host_hba_collector	6105	inventory	3,600 s	60 s
VMware ESXi Host Storage Paths	host_storage_path	vmware_host_storage_path_collector	6106	inventory	3,600 s	60 s
VMware ESXi Host Network Adapters	host_network_adapter	vmware_host_network_adapter_collector	6107	inventory	3,600 s	60 s
VMware ESXi Host Standard Switches	host_standard_switch	vmware_host_standard_switch_collector	6108	inventory	3,600 s	60 s
VMware ESXi Host Firewall Rules	host_firewall_rule	vmware_host_firewall_rule_collector	6109	inventory	3,600 s	60 s
VMware ESXi Host Services	host_service	vmware_host_service_collector	6110	inventory	3,600 s	60 s
VMware Datastores	datastore	vmware_datastore_collector	6111	inventory	3,600 s	60 s
VMware Distributed Switches	distributed_switch	vmware_distributed_switch_collector	6112	inventory	3,600 s	60 s
VMware Distributed Port Groups	distributed_port_group	vmware_distributed_port_group_collector	6113	inventory	3,600 s	60 s
VMware Resource Pools	resource_pool	vmware_resource_pool_collector	6114	inventory	3,600 s	60 s
VMware Folders	folder	vmware_folder_collector	6115	inventory	3,600 s	60 s
VMware ESXi Host Performance	host_stats	vmware_host_stats_collector	6151	metric	60 s	20 s
VMware ESXi Host CPU Usage	host_processor_stats	vmware_host_processor_stats_collector	6152	metric	60 s	20 s
VMware ESXi Host Memory Usage	host_memory_stats	vmware_host_memory_stats_collector	6153	metric	60 s	20 s
VMware ESXi Host Power Usage	host_power_stats	vmware_host_power_stats_collector	6154	metric	60 s	20 s
VMware ESXi Host Disk I/O	host_disk_stats	vmware_host_disk_stats_collector	6155	metric	60 s	20 s
VMware ESXi Host Storage Adapter I/O	host_hba_stats	vmware_host_hba_stats_collector	6156	metric	60 s	20 s
VMware ESXi Host Storage Path I/O	host_storage_path_stats	vmware_host_storage_path_stats_collector	6157	metric	60 s	20 s
VMware Datastore I/O	datastore_stats	vmware_datastore_stats_collector	6158	metric	60 s	20 s
VMware ESXi Host Network Traffic	host_network_adapter_stats	vmware_host_network_adapter_stats_collector	6159	metric	60 s	20 s
VMware Virtual Machines	vm	vmware_vm_collector	6200	inventory	3,600 s	60 s
VMware Virtual Machine Disks	vm_disk	vmware_vm_disk_collector	6201	inventory	3,600 s	60 s
VMware Virtual Machine Guest Partitions	vm_guest_partition	vmware_vm_guest_partition_collector	6202	inventory	3,600 s	60 s
VMware Virtual Machine CD/DVD Drives	vm_dvd_drive	vmware_vm_dvd_drive_collector	6203	inventory	3,600 s	60 s
VMware Virtual Machine Network Adapters	vm_network_adapter	vmware_vm_network_adapter_collector	6204	inventory	3,600 s	60 s
VMware Virtual Machine Snapshots	vm_snapshot	vmware_vm_snapshot_collector	6205	inventory	3,600 s	60 s
VMware Virtual Machine NPIV WWNs	vm_vhba	vmware_vm_vhba_collector	6206	inventory	3,600 s	60 s
VMware Virtual Machine Status	vm_status	vmware_vm_status_collector	6250	metric	60 s	10 s
VMware Virtual Machine Performance	vm_stats	vmware_vm_stats_collector	6251	metric	60 s	20 s
VMware Virtual Machine CPU Usage	vm_processor_stats	vmware_vm_processor_stats_collector	6252	metric	60 s	20 s
VMware Virtual Machine Memory Usage	vm_memory_stats	vmware_vm_memory_stats_collector	6253	metric	60 s	20 s
VMware Virtual Machine Disk I/O	vm_disk_stats	vmware_vm_disk_stats_collector	6254	metric	60 s	20 s
VMware Virtual Machine Network Traffic	vm_network_adapter_stats	vmware_vm_network_adapter_stats_collector	6255	metric	60 s	20 s

Every dataset takes these properties:

Field	Required	Type	Default	Description
interval	N	int	per dataset	Seconds between readings. At least the dataset's minimum above.
initial_lookback	N	int	the device's	Events and tasks only. Seconds before the first reading where a dataset with no checkpoint starts. 0 starts at the first reading.
pipeline_name	N	string	the device's	The preprocessing pipeline the dataset's records enter.

Wizard fields

Every VMware vSphere dataset's Dataset configuration step says how the dataset is read and has an Interval (Optional) field, in seconds: left empty, the dataset is read at its default; otherwise at least its minimum.

Rules

• Intervals. An interval under the dataset's minimum is left out with a warning in the device's log, and the rest of the device runs. Inventories are read hourly by default: what a host, a vCenter, and a virtual machine have changes rarely, and a part that fails is also an event the host logs, which the event dataset reads every minute. A performance counter dataset may be read every 20 seconds at most, because ESXi samples its counters every 20 seconds.
• One of each. A dataset type is read once per device, however many datasets of it are assigned. The first decides its settings; the others are left out with a warning.
• One model each. A dataset reads its model and nothing else.
• Pipelines. A dataset's records enter the pipeline the dataset selects, otherwise the device's. Two datasets on one device can reach two pipelines.
• Live capture names the dataset each record came from.

Inventories

Inventory datasets write every item at every reading. From a dataset's second reading on, each record carries event.action:

event.action	event.type	Meaning
added	creation	Not in the last delivered reading
changed	change	Its fields differ from the last delivered reading's
unchanged	info	As it was
removed	deletion	Gone: written once, as it was last seen, dated now

• The baseline is the last delivered reading. A reading that fails, or whose records could not all be written, never becomes the one the next compares with, so nothing is marked removed because a request failed.
• An item that could not be read is kept. A host or a machine whose properties the server could not all return is left out of the reading, with a warning, and is not marked removed; neither are its parts.
• What moves while an item stays the same is written but not compared, such as a sensor's reading, a datastore's free space, a machine's power state, or a partition's free space. Each table below says which fields these are.
• The first reading after a start marks nothing: the baseline is kept in memory.

ESXi Hosts

host is the node dataset: a vCenter device writes every host it manages, and a standalone ESXi device the host itself. A host is known by its managed object id (host-1234; ha-host on a standalone host).

Field	Holds
host.*	name, id (the system UUID), ip (the VMkernel addresses), mac (the physical NICs'), os.name (VMware ESXi), os.version, os.full
vmware.host.*	id (managed object id), name, product.name/full_name/version/build, boot_time, manufacturer, model, serial_number, uuid, management_server_ip (the vCenter managing it), reboot_required, cpu.model/mhz/sockets/cores/threads, memory.size, power_policy (dynamic, static, low, custom)

A host moved to another cluster, or whose cluster was renamed, is changed; so is one that rebooted, which has a new boot_time.

What an ESXi Host Has

Ten datasets read the parts of every host, one record per part. Every record names its host (host.name, vmware.host.id, vmware.host.name) and carries the host's datacenter and cluster. Fields sit under vmware.<dataset>.

Dataset	One record per	Fields
host_processor	Processor package (socket)	index, description (the model), vendor, mhz, bus_mhz, cores, threads, thread_ids (the logical processors host_processor_stats names), status (green, yellow, red, unknown)
host_fan	Fan sensor	id, name, health (green, yellow, red, unknown), reading in its units, rate_units, timestamp; from vSphere 8.0 Update 1, fru.type/part_name/part_number/manufacturer/serial_number. The reading and its time are not compared.
host_power_supply	Power sensor	As host_fan
host_sensor	Hardware sensor, every type	As host_fan, and type (temperature, voltage, fan, power, memory, storage, systemBoard, …)
host_hba	Storage adapter (vmhba), every kind	device, type (block, sas, parallel_scsi, fibre_channel, fcoe, iscsi, nvme_tcp, nvme_rdma, pcie), model, driver, pci, bus, status, storage_protocol, and per kind its world wide names, iSCSI name, speed, or associated NIC
host_storage_path	Path from an adapter to a LUN	name (vmhba1:C2:T0:L0), adapter, lun (its canonical name), lun_number, state (active, standby, disabled, dead), working, policy (VMW_PSP_FIXED, VMW_PSP_RR, VMW_PSP_MRU), preferred, array_type_policy, and Fibre Channel or iSCSI target details
host_network_adapter	Physical NIC (vmnic) or VMkernel adapter (vmk)	name, type (physical, vmkernel), mac; a physical NIC's driver, driver_version, firmware_version, pci, link, speed, full_duplex; a VMkernel adapter's ip, netmask, ipv6, dhcp, mtu, portgroup, netstack, and its standard or distributed switch
host_standard_switch	Standard switch	name, ports, ports_available, mtu, uplinks, portgroups, and its link discovery, security, offload, shaping, and teaming policies
host_firewall_rule	Rule of a firewall ruleset	ruleset, label, service, enabled, required, allowed_hosts.all/ip/networks, direction, protocol, port, end_port, port_type
host_service	Service	key (TSM-SSH), label, policy (on, off, automatic), running, required, uninstallable, rulesets, package

What a vCenter Holds Beside Its Hosts

Five datasets read the objects a vCenter holds that belong to no single host. They are tagged with their datacenter; no cluster tags them. A standalone ESXi host has its own datastores, root resource pool, and folders, and no distributed switches.

Dataset	One record per	Fields
datastore	Datastore	id, name, type (VMFS, NFS, NFS41, vsan, VVOL, …), url, capacity, free_space (not compared), accessible, multiple_host_access, maintenance_mode, overall_status, its VMFS, NFS, or vVol details, luns (not compared), and hosts (each mounting host's id, name, access_mode, mounted, accessible). A datastore one host alone mounts is also tagged with that host.
distributed_switch	Distributed switch	id, name, uuid, description, overall_status, product.*, host_count, hosts, portgroups, ports (not compared), max_ports, max_mtu, uplinks, link_discovery.*, config_version, create_time, default_port.*
distributed_port_group	Distributed port group	id, key, name, switch.id/name, type (earlyBinding, ephemeral), ports, auto_expand, uplink, host_count, vm_count (not compared), overrides_allowed.*, and its ports' settings: vlan.id, vlan.trunk or vlan.pvlan_id, shaping, teaming, security, and LACP
resource_pool	Resource pool or vApp	id, name, type (ResourcePool, VirtualApp), path, owner.*, parent.*, cpu.reservation.mhz, cpu.limit.mhz, memory.reservation.bytes, memory.limit.bytes, unlimited, expandable_reservation, shares, shares_level, overall_status. A cluster's pool is tagged with the cluster, a standalone host's with the host.
folder	Folder of the inventory	id, name, child_types, parent.*, path (/Amsterdam/vm/web)

Virtual Machines

vm writes every virtual machine, templates included, one record each. A virtual machine is not a device of its own: it is named under vmware.vm (id, the managed object id vm-1234, and name), and host.name is the ESXi host it runs on.

Field (under vmware.vm)	Holds
id, name, uuid, instance_uuid	Its managed object id, name, BIOS UUID, and vCenter instance UUID
template, version, firmware	Whether it is a template, its hardware version (vmx-19), and its firmware (bios, efi)
os.id, os.full_name	The guest operating system it is configured for
notes, created	Its annotation, and when it was created (vSphere 6.7 and later)
path, folder.*, vapp.*, resource_pool.*	Where it sits in the inventory
cpu.*, memory.*	Its virtual CPUs and memory, and their reservations, limits, and shares
files.*, datastores	Its files, and each datastore they are on
tools.*	Its VMware Tools settings
power_state, connection_state	Not compared
guest.hostname, guest.ip, guest.os.*, guest.tools.version	While VMware Tools runs. Not compared.

A machine moved to another cluster, datacenter, folder, or resource pool is changed, as is one given more memory or CPUs, renamed, or made a template. Its ESXi host is not compared: DRS moves a cluster's machines on its own, and the record says where the machine runs now. What the guest says of itself comes through VMware Tools, so a machine whose Tools does not run has none of it written.

What a Virtual Machine Has

Six datasets read the parts of every virtual machine, one record per part. Every record names its machine (vmware.vm.id, vmware.vm.name) and the host it runs on. A powered-off machine keeps its parts.

Dataset	One record per	Fields
vm_disk	Virtual disk, every kind	key, label, instance (scsi0:1, which its performance counters name it by), controller.*, unit_number, capacity, backing, file, datastore.*, disk_mode, provisioning (thin, thick_lazy_zeroed, thick_eager_zeroed), rdm.*, iops.*, shares, shares_level; while the machine has snapshots, current_file and delta_disks (not compared)
vm_guest_partition	Partition the guest reports, while VMware Tools runs	path (C:\, /boot), capacity, free_space, used_space (not compared), used_pct, filesystem, disks (vSphere 7.0 and later)
vm_dvd_drive	CD/DVD drive	key, label, controller.*, unit_number, media.type (iso, host_device, client_device, none, …), media.file, media.datastore.*, start_connected, allow_guest_control, connected, status
vm_network_adapter	Network adapter	key (4000, which its performance counters name it by), label, type (vmxnet3, e1000e, …), mac, address_type, connected, status, network.type (standard, distributed, opaque), network.name, its port group and switch, network.vlan.*; while VMware Tools runs, guest.ip, guest.connected, guest.network
vm_snapshot	Snapshot	id, name, description, created, power_state, quiesced, replay_supported, parent.id/name, current (not compared)
vm_vhba	Machine given world wide names (NPIV)	type (vc, host, external), node_wwns, port_wwns, desired_node_wwns, desired_port_wwns, disabled, on_non_rdm_disks

Status

host_status and vm_status write one record per host or machine at every reading, dated at the reading. status is 1 when the host or machine is up, otherwise 0, with the state that says why in reason.

Dataset	Fields
host_status	vmware.host.id/name; vmware.host_status.status (1 when connected and powered on), reason (host connection state: disconnected), connection_state (connected, disconnected, notResponding), power_state, maintenance_mode, overall_status. A host in maintenance mode is up.
vm_status	Every machine but the templates. vmware.vm_status.status (1 when connected and powered on), reason (vm power state: poweredOff), power_state, connection_state, overall_status, config_status, guest_heartbeat_status, tools.running_status/version_status/status, boot_time and uptime while it runs, suspend_time, consolidation_needed, fault_tolerance_state

Performance Counters

The performance counter datasets read the latest realtime sample of every connected, powered-on host or running virtual machine. ESXi samples its counters every 20 seconds and keeps about an hour of them.

• Dated when the sample ended. A record's @timestamp is when its sample ended, and vmware.<dataset>.interval is its length, 20. A sample is written once, so at the default interval of 60 seconds a reading writes one sample in three.
• Counters are resolved by name, against each server's own catalogue. A counter a host or machine does not sample is left out, never written as zero.
• One record per instance or per host. A counter the host keeps per CPU, disk, NIC, adapter, path, or datastore is one record per instance, which vmware.<dataset>.instance names.
• Units. Percentages are fractions of 1, sizes are bytes, and rates are bytes per second; the rest are written as the server counts them: MHz, milliseconds (ms), microseconds (us), watts, joules, counts during the sample (count), and per-second averages (per_sec).
• A host or machine that is not running samples nothing and is left out; its state is its status dataset's.

ESXi Hosts

Dataset	One record per	Counters	instance
host_stats	Host	The host-wide aggregate of the cpu, net, disk, datastore, storageAdapter, and storagePath counters, under cpu., net., disk., datastore., storage_adapter., and storage_path.; and cpu.sockets, cpu.cores, cpu.threads, memory.free.bytes, memory.free.pct	none
host_processor_stats	Logical processor	cpu, those the host keeps per processor	Its number: one of host_processor's thread_ids
host_memory_stats	Host	mem	none
host_power_stats	Host	power: power.watts, power_cap.watts, energy.joules	none
host_disk_stats	Disk (LUN)	disk, those the host keeps per disk	Its canonical name: a host_storage_path's lun
host_hba_stats	Storage adapter	storageAdapter	Its device, vmhba1: host_hba's device
host_storage_path_stats	Path to a LUN	storagePath	Its transport name, ending in the LUN's canonical name
datastore_stats	Datastore a host mounts	datastore, and the datastore's space: capacity.bytes, free.bytes, used.bytes, used.pct	The UUID its URL ends in; vmware.datastore.id and name name the datastore
host_network_adapter_stats	Physical NIC	net	Its device, vmnic0: host_network_adapter's name

Virtual Machines

Dataset	One record per	Counters	instance
vm_stats	Running machine	The machine-wide aggregate of its net and virtualDisk counters, under net. and virtual_disk.; cpu.count, memory.size.bytes, storage.committed.bytes, storage.uncommitted.bytes, storage.unshared.bytes; while VMware Tools runs, guest.capacity.bytes, guest.free.bytes, guest.used.bytes, guest.used.pct	none
vm_processor_stats	Running machine	cpu, the machine's own	none
vm_memory_stats	Running machine	mem, the machine's own	none
vm_disk_stats	Virtual disk of a running machine	virtualDisk	scsi0:1: the disk's vm_disk instance
vm_network_adapter_stats	Network adapter of a running machine	net	Its device key, 4000: its vm_network_adapter key

A machine's network counters also name the host's physical NICs its traffic crossed; those belong to host_network_adapter_stats and are left out.

Events and Tasks

event and task read what happened since their checkpoint, which is kept in the Director's cluster-replicated store, so a restart or a failover resumes from it.

• Written after delivery. The checkpoint is stored only once the records it covers are delivered: a crash before then reads them again, so a record is written at least once and never lost.
• The first reading of a dataset with no checkpoint starts initial_lookback seconds back: the dataset's, else the device's, else a day.
• A long history is read a share at a time, at most 200,000 events or tasks per reading; the dataset catches up over the next readings.

Dataset	Fields
event	message (the server's formatted message), event.id (the event's key), event.code (its type, VmPoweredOnEvent, or an extended event's id), log.level, user.name, host.name; vmware.event.key, chain_id, type, category (info, warning, error, user), datacenter/compute_resource/host/vm/datastore/network/dvs (each id and name), task.id, arguments
task	message (Power On virtual machine - web01), event.id (the task's key), event.action (VirtualMachine.powerOn), event.type (info queued, start running, end finished), event.outcome, event.start, event.end, event.duration, user.name, host.name; vmware.task.key, name, description_id, label, state (queued, running, success, error), cancelled, progress, entity.type/id/name, queue_time, start_time, complete_time, queued_ms, execution_ms, error, reason (user, system, alarm, schedule)

A task is written each time its state changes: when it is first seen queued or running, and when it ends. A standalone ESXi host keeps no task history, so its tasks are read from its recent tasks alone: a task that ended while the device was down for longer than the host keeps its recent tasks (minutes) is missed.

Examples

Reading a vCenter's hosts and virtual machines hourly, their status every minute, the hosts' CPU every 20 seconds, and the events with a week's lookback...

- id: 512300130 name: vc01 type: vmware status: true properties: address: "vcsa.example.com" username: "svc-virtualmetric@vsphere.local" password: "$secret{store=corp-vault,ref=vmware/vcenter}" definitions: - name: vmware_host_collector status: true - name: vmware_host_status_collector status: true - name: vmware_host_processor_stats_collector status: true properties: interval: 20 - name: vmware_vm_collector status: true - name: vmware_vm_status_collector status: true - name: vmware_event_collector status: true properties: initial_lookback: 604800