Skip to main content

VMware Cloud Director Datasets

VMware Cloud Director datasets define what a VMware Cloud Director device reads from a Cloud Director (dataset type vcloud). They attach to devices of type vcloud, and in Datasets & Profiles the category is labelled VMware Cloud Director. To create and assign datasets through the web interface, see Management. For the dataset and profile model, see Overview.

Collector Types​

There are eight datasets, one per data model, each with its own definition id, event.dataset, interval, and pipeline, so it can be searched, routed, and processed on its own. Seven are the cloud's own, with event.dataset vcloud.<dataset>. The eighth, the ESXi hosts, is the VMware vSphere host dataset.

Dataset (console title)DatasetDefinitionDefinition idKindDefault intervalMinimum
vCloud Organizationsorganizationvcloud_organization_collector8100inventory3,600 s60 s
vCloud Usersorganization_uservcloud_organization_user_collector8101inventory3,600 s60 s
vCloud vCenter Serversvcentervcloud_vcenter_collector8102inventory3,600 s60 s
vCloud VDCsvdcvcloud_vdc_collector8103inventory3,600 s60 s
vCloud vAppsvappvcloud_vapp_collector8104inventory3,600 s60 s
vCloud Virtual Machinesvmvcloud_vm_collector8105inventory3,600 s60 s
vCloud Eventseventvcloud_event_collector42event log60 s10 s
VMware ESXi Hostsvmware.hostvmware_host_collector6100inventory3,600 s60 s

vcenter and vmware_host_collector read the provider's view, so only a device that logs in to the System organization as a System administrator fills them. On a device that logs in to any other organization they read nothing, and the device's message says why. See the Account section of VMware Cloud Director.

Every dataset takes these properties:

FieldRequiredTypeDefaultDescription
intervalNintper datasetSeconds between readings. At least the dataset's minimum above.
pipeline_nameNstringthe device'sThe preprocessing pipeline the dataset's records enter.
initial_lookbackNintthe device'sevent only. Seconds before its first reading where the dataset starts when it has no checkpoint. 0 starts at the first reading.

Wizard fields​

Every VMware Cloud Director dataset's Dataset configuration step says how the dataset is read and has an Interval (Optional) field, in seconds: left empty, the dataset is read at its default; otherwise at least its minimum.

Rules​

  • Intervals. An interval under the dataset's minimum is left out with a warning in the device's log, and the rest of the device runs. The inventories are read hourly by default, the audit trail every minute.
  • One of each. A dataset type is read once per device, however many datasets of it are assigned. The first decides its settings; the others are left out with a warning.
  • Pipelines. A dataset's records enter the pipeline the dataset selects, otherwise the device's. Two datasets on one device can reach two pipelines.

Inventories​

Inventory datasets write every item at every reading. From a dataset's second reading on, each record carries event.action:

event.actionevent.typeMeaning
addedcreationNot in the last delivered reading
changedchangeIts fields differ from the last delivered reading's
unchangedinfoAs it was
removeddeletionGone: written once, as it was last seen, dated at the reading
  • The baseline is the last delivered reading. A reading that fails, or whose records could not all be written, never becomes the one the next compares with, so nothing is marked removed because a request failed.
  • Unread is not removed. An organization whose administrative record the server refuses, or that is gone since the list named it, is left out of the reading and kept as the last reading held it, and so are its users and VDCs. A user, a VDC, or an ESXi host whose own record cannot be read is kept the same way, and so is a VDC one of whose storage profiles cannot be read.
  • What moves while an item stays the same is written but not compared, such as a virtual machine's power state or a VDC's use. Each table below says which fields these are. A machine that moved to another ESXi host did not change.
  • The first reading after a start marks nothing: the baseline is kept in memory.

Organizations and Users​

DatasetOne record perFields
organizationOrganizationvcloud.organization.id, name, full_name, description, enabled
organization_userUser of an organizationuser.id, user.name, user.full_name, user.email, user.domain (the organization the user logs in to), user.roles; vcloud.organization_user.id, name, full_name, email, telephone, description, name_in_source, provider_type (INTEGRATED, LDAP, SAML, …), role.id/name, enabled, locked, external, group_role, stored_vm_quota, deployed_vm_quota (0 is no quota)

vCenter Servers​

DatasetOne record perFields
vcentervCenter Server attached to the cloudvcloud.vcenter.id, name, url, version, uuid, enabled, supported, busy, status (READY, …), listener_state (CONNECTED, DISCONNECTED, …). Not compared: busy, status, listener_state.

VDCs, vApps, and Virtual Machines​

DatasetOne record perFields
vdcOrganization VDCvcloud.vdc.id, name, description, allocation_model (AllocationPool, AllocationVApp, ReservationPool, Flex), enabled, status (1 ready, 0 not ready, -1 could not be created), cpu.allocated/reserved/used/limit (each .mhz), memory.allocated/reserved/used/limit (each .bytes), storage.used/limit (.bytes, the sum of the VDC's storage profiles), cpu.unlimited, memory.unlimited, storage.unlimited (an unlimited resource has no limit written), nic_quota, network_quota, used_network_count, vm_quota (0 is no quota), resource_pool.id (its vSphere resource pool). Not compared: cpu.used, memory.used, storage.used, used_network_count.
vappvApp, except those the Cloud Director makes for a standalone virtual machinevcloud.vapp.id, name, status (POWERED_ON, POWERED_OFF, SUSPENDED, …), owner, created, busy, deployed, enabled, expired (its lease), maintenance_mode, public, auto_delete_notified, auto_undeploy_notified, honor_boot_order, vm_count, cpu.count, cpu.allocation.mhz, memory.allocation.bytes, storage.bytes. Not compared: status, busy, deployed, expired, and the lease notices.
vmVirtual machine, vApp templates' includedvcloud.vm.id, name, moref (its managed object id in its vCenter Server), status, description, guest_os, hardware_version, cpu.count, memory.size, storage.allocated, network, ip_address, storage_profile, datastore, catalog, created, busy, deployed, published, template, deleted, maintenance_mode, expired, standalone (the Cloud Director made its vApp for it), tools.status/version. Not compared: status, busy, deployed, expired, ip_address, tools.status.

A vcloud.vm record is the machine as the Cloud Director holds it, with its vApp, VDC, organization, storage profile, and catalog, keyed by the cloud's own id. A VMware vSphere device's vmware.vm record is the same machine as vSphere holds it, keyed by its managed object id. A Cloud Director device writes no vmware.vm record: vcloud.vm.moref, with vcloud.vcenter, joins the two.

ESXi Hosts​

vmware_host_collector reads every ESXi host of every vCenter Server attached to the cloud, through the provider's extension API, into the VMware vSphere host dataset: the same definition id, event.dataset (vmware.host), and fields as on a VMware vSphere device, so an ESXi host is one model whichever device read it. The host records carry no vcloud.* field.

The Cloud Director knows less of a host than vSphere does, and the record holds only what it knows:

DatasetOne record perFields
vmware.hostESXi hosthost.name, host.os.name, host.os.version; vmware.host.id (its managed object id), name, product.name/version (VMware ESXi, 8.0.2), cpu.model, cpu.sockets, memory.size

Hardware, serial number, boot time, addresses, CPU speed, cores and threads, and the datacenter and cluster are left out rather than guessed. A host is keyed by the cloud's own id of it, since two vCenter Servers may both have a host-12.

Events​

event reads the Cloud API's audit trail: every organization's events for a System administrator, the organization's own for its administrator. The events are written in the order they happened, each once, dated when it happened. The Cloud Director keeps its audit trail for 45 days by default.

DatasetOne record perFields
eventAudit eventmessage (the event's description), event.id (its URN), event.action (its type, com/vmware/cloud/event/vm/deploy), event.outcome (success, failure), user.id, user.name; vcloud.event.id, type, service_namespace, status (SUCCESS, FAILURE), entity.id/type/name (what it happened to), task.id, cell_id, external, properties (its additional properties, currentContext.user.clientIpAddress among them)

The dataset resumes from where the last delivered reading stopped:

  • Across a restart or a failover. The checkpoint is kept in the Director cluster's replicated store, per device, so a restart, or another node after a failover, resumes from it.
  • Written after delivery. A reading's checkpoint is stored only once its events are delivered. A crash before then reads those events again at the next start: at least once, never lost.
  • The first reading of a dataset with no checkpoint goes back as far as the initial lookback, a day unless the dataset or the device says otherwise.
  • A long trail is read a share at a time. A reading reads at most 200 pages of 128 events, and the next reading continues from where it stopped.

Examples​

Reading the virtual machines every 15 minutes into their own pipeline, and the audit trail from an hour back...

- id: 512300140
  name: vcd01
  type: vcloud
  status: true
  properties:
    address: "vcloud.example.com"
    username: "svc-virtualmetric"
    password: "$secret{store=corp-vault,ref=vcloud/provider}"
  definitions:
    - name: vcloud_vm_collector
      status: true
      properties:
        interval: 900
        pipeline_name: "vcloud-vms"
    - name: vcloud_event_collector
      status: true
      properties:
        initial_lookback: 3600
    - name: vmware_host_collector
      status: true