Clusters

Enterprise

Clusters group multiple Directors together for high availability. When one Director fails, the remaining Directors automatically take over its workload. Agents and Devices connect to the cluster as a whole, ensuring continuous operation as long as at least one Director remains healthy.

Provider → Device → Preprocessing → Pipeline → Postprocessing → Target → Consumer

Cluster (groups Directors for high availability and automatic failover)

Permanent Assignment

Once a Director is added to a cluster, it cannot be individually modified, updated, or removed. All configuration changes must be made at the cluster level. The only way to release a Director from a cluster is to delete the entire cluster, after which the Director returns to standalone status.

Access

The Directors page displays two tabs: Directors and Clusters. Click the Clusters tab to manage cluster configurations. The Clusters tab requires CLUSTER_READ permission.

Overview

The Clusters table displays all configured clusters with the following columns:

Column	Description
Cluster name	Cluster identifier
Mode	Management type (Managed or Self-managed)
Director count	Number of Directors in the cluster
Log stream	Log stream indicator
Cluster status	Operational state (Enabled/Disabled)
Director connection status	Connection state of member Directors
Actions	Menu for cluster operations

Table Controls:

Filter clusters using Search clusters (by name) and Status dropdown (All, Enabled, Disabled). The Refresh button reloads table data. Click Create new cluster to launch the creation wizard (requires CLUSTER_CREATE permission).

Create Cluster Wizard

The cluster creation process guides you through a 5-step wizard: General settings, Access management, TLS configuration, Update management, and Review and create.

General Settings

Basic Configuration:

1. Cluster name - Unique identifier for the cluster
2. Port - Communication port for cluster coordination (default: 8081)
3. Mode - Management approach, selected under Choose mode:
   • Managed: VirtualMetric manages cluster infrastructure
   • Self-managed: User controls cluster deployment and configuration. A self-managed cluster is always on the Manual update policy — see Manual Update

Director Selection:

Select Directors to include in the cluster. Requirements:

• Minimum 3 Directors, maximum 15
• Must be an odd number (3, 5, 7...) for quorum
• Directors must be stand-alone and unconfigured. Directors that cannot be added: Directors already assigned to another cluster, Directors with an existing configuration (tokens, devices, or targets), Directors set to orchestrated mode, and Directors set to serverless mode

For each Director, specify:

• Director - Select from available Directors dropdown
• IP - Auto-populated from Director configuration
• Port - Communication port (auto-populated, editable)

note

Directors already assigned to a cluster or previously configured are not available for selection.

Access Management

Access restriction:

Restrict which IP addresses can connect to the cluster:

• Access restriction status - Toggle to enable IP filtering
• Type:
  • Single IP/IP block - Single IP address or IP block
  • IP range - IP address range (start-end)
  • CIDR - CIDR notation (e.g., 10.0.0.0/8)
• Trusted sources - List of allowed IP addresses/ranges

Access tokens:

Create authentication tokens for Agents and Devices connecting to the cluster:

• Access tokens - Toggle to require token authentication
• Token Name - Identifier for the token
• Token Value - Auto-generated when token is created (use copy button to copy)
• Expiration - Token validity period (default: 3 months):
  • 1 month
  • 3 months
  • 1 year
  • Unlimited

TLS Configuration

Select how TLS is managed for agent-to-cluster communication using radio tiles:

• Auto-generated - The cluster manages its own certificate authority (CA) and certificates
• Custom certificate - The cluster uses your certificate and private key
• TLS offloading - TLS termination is handled by your load balancer

This step configures the cluster's HTTPS listener — the same proxy_tls settings a standalone Director carries, applied across every node. It does not configure encryption between cluster members; that is a separate concern, set after creation on the Configuration tab.

The fields match the Director wizard, with cluster-specific labels (Cluster Address for Agent Connections, Cluster HTTP port) and one difference: the Custom certificate mode offers no discovery endpoint control. See Transport Layer Security for the full field reference.

Update Management

Configure the cluster's update policy. By default the organization-level update settings apply; enable the Custom update policy toggle to override them for this cluster. See Updates for the policy options.

Review and Create

A read-only summary of the values entered in the preceding steps. Click Create cluster to complete cluster creation.

Detail View

Clicking a cluster from the overview table opens the detailed management interface with eight tabbed panels: Cluster overview, Configuration, Access management, TLS configuration, Update management, Live data, Console logs, and Activity logs.

Cluster Overview Tab

View cluster configuration and Director membership:

Cluster information:

• Name - Cluster identifier
• Mode - Managed or Self-managed
• Port - Cluster communication port
• Status - Enabled or Disabled

Directors Table:

Lists all Directors in the cluster with:

• Name - Director identifier
• IP address - Director network address
• Port - Director communication port
• Status - Connection status (Connected/Not connected)

Configuration Tab

View and edit inter-node TLS — encryption between the members of the cluster:

• TLS encryption - Enabled or Disabled
• Certificate - Certificate content (required when enabled)
• Private key - Private key content (required when enabled)

Both values are masked. Click Manage cluster configuration, then Change TLS certificate details to edit them. Cancel editing TLS discards any edits and restores the masked view. The private key is never displayed, and a stored key is kept unless you replace it. Turning TLS encryption off and saving removes the stored private key.

note

This is not the same as the TLS configuration tab. Inter-node TLS secures traffic between cluster members and is available only after the cluster exists — the creation wizard does not set it. The TLS configuration tab carries the cluster's HTTPS listener settings for agent connections, described in Transport Layer Security.

Access Management Tab

View and edit access controls:

Access restriction:

• Current restriction status and configured sources
• Edit to add, remove, or modify IP restrictions

Access tokens:

• List of configured tokens with expiration dates
• Create new tokens or revoke existing tokens

Update Management Tab

Configure the cluster's update policy — see Updates for the available settings.

Live Data Tab

Capture pipeline messages flowing through the cluster at configurable processing stages — see Live Data.

Console Logs Tab

Stream real-time system debug messages from the cluster — see Console Logs.

Cluster Activity Logs Tab

Track cluster operations and lifecycle events:

Column	Description
User	Email of user who performed action
User IP	IP address from which action was performed
Object Type	Always "Cluster" for this view
Object	Name of affected cluster
Action	Operation performed
Action Description	Detailed description
Created At	Timestamp of action

Action Types:

• ClusterCreated - New cluster created
• ClusterUpdated - Cluster configuration modified
• ClusterDeleted - Cluster removed

Cluster Operations

Enable / Disable

To change cluster status:

1. Navigate to cluster detail view or use Actions menu from overview
2. Click Actions menu
3. Select Enable cluster or Disable cluster
4. Confirmation notification confirms status change

Disabling a cluster also disables all member Directors. Re-enabling restores previous Director states.

Download Config

For Self-managed Clusters:

Download the cluster configuration file:

1. Navigate to cluster detail view
2. Click Actions menu
3. Select Download config
4. Deploy downloaded configuration to each Director in the cluster

Export Template

1. Navigate to cluster detail view
2. Click Actions menu
3. Select Export template

The cluster opens as a Bicep template, to deploy again or to change first. See Export Template.

Delete

Delete Cluster Process:

1. Navigate to cluster detail view or use Actions menu
2. Click Actions menu
3. Select Delete cluster
4. Confirm deletion in modal dialog

warning

Clusters with assigned Devices cannot be deleted. Reassign or remove all Devices from cluster Directors before deletion.

Deleting a cluster also removes all Director associations. The Directors themselves are not deleted but return to standalone status.

Manual Update

On a cluster whose update policy is Manual, Download update hands out one download: the Director build for the operating system and architecture that the cluster's Directors report. It is offered from the cluster's row actions on the Clusters overview, and from the update banner on the cluster's detail page. Install that build on each Director of the cluster.

Nothing is triggered on the platform: no update status opens and no configuration is deployed, as with a standalone Director's Manual update.

This applies to:

• Every cluster on the Manual policy.
• Every self-managed cluster, which is always on Manual. The platform refuses Auto for it: "self-managed clusters cannot use automatic updates".

Serverless Directors cannot join a cluster, so a cluster's update is always a download, never a container image roll. Auto mode is unchanged: it still needs every Director of the cluster connected.

What Download update answers:

The cluster's Directors	Result
All report the same system	The build for that system
Some report a system; others have been disconnected long enough to report none	The build for the system the reporting ones run on
They report different systems	Refused, naming the link to each system's build: install each Director's own
None reports a system, and none is connected	Refused with DIRECTOR_NOT_CONNECTED: "No director of the cluster is connected…". Try again once one reconnects
None reports a system, but one is connected	Fails: no Director has reported its operating system and architecture

A Director's reported system outlives its connection for a while, so a cluster whose Directors went away recently still gets its download.

note

A cluster whose Directors run on different systems has no single build. Installing one system's build on every Director would break the others, which is why the request is refused rather than answered with one of them.

Auto Update

Cluster Auto Update coordinates version updates across all member Directors. When configured at the cluster level, updates propagate to all Directors while maintaining cluster availability.

Cluster Update Configuration

Configure Auto Update in the cluster configuration to apply settings uniformly across all member Directors:

Property	Type	Required	Description
mode	string	Y	Set to auto to enable automatic updates
method	string	Y	Update trigger method: immediate or schedule
version	string	Y	Target version for all cluster Directors
cron	string	N	Cron expression for scheduled updates (minimum 30 seconds)
interval	duration	N	Fallback polling interval (default: 30s minimum)

Example Cluster Update Configuration:

update:
  mode: auto
  method: schedule
  version: "2.5.0"
  cron: "0 2 * * 0"  # Every Sunday at 2:00 AM

Rolling Update Behavior

Cluster updates follow a rolling update pattern to maintain availability:

1. Quorum Preservation - Updates proceed only while quorum is maintained
2. Sequential Updates - Directors update one at a time, not simultaneously
3. Health Verification - Each Director must report healthy before the next update begins
4. Automatic Failover - Workloads shift to remaining Directors during individual updates

note

During a rolling update, cluster capacity is temporarily reduced. Schedule updates during low-traffic periods for production clusters.

Cluster Health Monitoring

Monitor cluster update progress through health reporting:

Field	Description
version	Current running version per Director
scheduled_version	Target version pending update
cluster_update_status	Overall cluster update progress

The cluster detail view displays version information for each member Director, allowing you to track update progress across the cluster.

Update Considerations

Minimum Cluster Size:

Rolling updates require at least 3 healthy Directors to proceed. Clusters at minimum size (3 Directors) have reduced fault tolerance during updates.

Mixed Version Operation:

During rolling updates, the cluster temporarily operates with mixed versions. DataStream maintains backward compatibility between adjacent versions to ensure seamless operation.

Rollback Behavior:

If a Director fails to update, it automatically rolls back to the previous version and rejoins the cluster. The cluster continues operating while the failed update is investigated.

warning

Do not update a cluster's Directors outside its update policy. On Auto, the cluster coordinates the update across its Directors. On Manual, download the update from the cluster and install it on each of its Directors.

Internal Update Timing

The following hardcoded constants govern cluster update behavior:

Constant	Value	Purpose
Update Check Interval	10 s	How often the cluster polls for pending updates
Node Timeout	30 min	Maximum wait for a single node to complete its update
Health Check Wait	30 s	Delay after update before health check runs
Leader Stability	30 s	Required leader stability before proceeding to next node
Max Retries	3	Maximum update retries per node. The offline-node retry path reuses the same limit — there is no separate offline budget
Command Retry Interval	30 s	Delay between retry attempts

For the cluster environment file format, see Environment Files. For the vmetric.yml service settings, see Service Configuration.