Service Configuration

This reference covers the configuration files and fields available for self-managed Director deployments. Managed Directors receive their configuration from the DataStream platform automatically; the fields documented here are relevant when you manage Director infrastructure directly.

For Director concepts and GUI-based setup, see Overview. For installation procedures, see Deployment.

The reference is split across four pages:

Page	Covers
Service Configuration (this page)	vmetric.yml — identity, network, queue, processing defaults, persistent storage
Environment Files	config/environments/*.yml — topology, clustering, ACL, TLS, VIP
Health and Limits	Health check thresholds and the hardcoded internal timing constants
Examples	Complete worked configurations

Configuration Files

A Director reads two categories of configuration at startup:

File	Location	Purpose
Service configuration	vmetric.yml in the Director working directory	Core service settings (identity, network, queue, processing defaults)
Environment configuration	config/environments/*.yml	Topology, clustering, ACL, and TLS

Supported formats: YAML (.yml / .yaml), TOML (.toml), and VMF (.vmf). Maximum file size: 1 MB.

Config file lookup order (first match wins):

1. vmetric.toml
2. vmetric.yml
3. vmetric.yaml

Debug Mode

When the virtualmetric=debug environment variable is set, the service loads vmetric-local.yml (or .toml / .yaml) instead of the standard file. This is intended for local development and troubleshooting only -- do not use in production.

---

Director Identity

Field	YAML Path	Type	Default	Description
ID	director.id	int64	--	Unique Director instance identifier
Status	director.status	bool	true	Enable or disable the Director service
Stream	director.stream	string	--	NATS stream name for Director replies

Network

Field	YAML Path	Type	Default	Description
Address	listen.address	string	0.0.0.0	Bind address for all listeners
External Address	listen.external_address	string	--	Advertised address for remote connections (comma-separated for multiple)
Port	listen.port	int	8890	Main collector listening port
WebSocket Port	listen.ws_port	int	9080	VMMQ WebSocket port
HTTP Config Port	listen.httpconfigport	int	80 / 443	HTTP configuration server port

Queue

Field	YAML Path	Type	Default	Description
Limit	queue.limit	int	172800 (48 h)	Queue size limit in seconds
Parallelism	queue.parallelism	int	CPU count	Queue processing parallelism
Non-Existing Target Limit	queue.non_existing_target_limit	int	86400 (24 h)	TTL for queued data to non-existing targets (seconds)
Cleanup Interval	queue.cleanup_interval	int	900 (15 min)	Cleanup interval for non-existing target queues (seconds)
DeQueue Interval	queue.dequeue_interval	int	1	DeQueue processing interval in seconds

Debug and Logging

Field	YAML Path	Type	Default	Description
Level	debug.level	int	0	Debug verbosity: 0 = off, 1--3 = increasing detail
Console Status	debug.console.status	bool	false	Enable console debug output
Log Status	debug.log.status	bool	false	Enable file-based debug logging
Remote	debug.remote	bool	false	Enable remote debug mode (disables device tasks when true)

Fleet

Fleet settings control how a Director connects to the DataStream cloud platform. Self-managed Directors that operate independently (no fleet connection) set self_managed: true and omit the fleet block.

Field	YAML Path	Type	Default	Description
Self-Managed	self_managed	bool	--	If true, Director runs independently without fleet connection
Fleet Type	fleet.type	string	--	Connection type (e.g., httpclient)
Fleet Instance	fleet.instance	string	--	Fleet environment instance name
Fleet Address	fleet.address	string	--	Fleet server WebSocket URL
Fleet Token	fleet.token	string	--	Authentication token for fleet

Data Processing Defaults

These defaults apply to all data processing unless overridden at the device or target level.

Field	YAML Path	Type	Default	Description
Compression	compression	bool	true	Enable data compression
Compression Level	compression_level	string	--	Compression level (algorithm-specific)
Reuse	reuse	bool	true	Enable connection reuse
Normalization	normalization	bool	true	Enable data normalization
Smart Data Engine	smart_data_engine	bool	true	Enable SDxL processing
No Buffer	no_buffer	bool	false	Disable buffering
Batch Size	batch_size	int64	10000	Read batch size
Write Batch Size	write_batch_size	int64	1000	Write batch size
Workers	workers	int	CPU count	Worker count
Chunk Size	chunk_size	int	4194304 (4 MB)	Data chunk size in bytes
Min Chunk Count	min_chunk_count	int	10	Minimum chunk count
Stats Enabled	stats_enabled	bool	false	Enable per-device, per-route, and per-target metrics collection. Fleet-managed Directors override this to true automatically.
Stats Drop On Full	stats_drop_on_full	bool	true	When the stats channel buffer is full, drop new metrics rather than blocking the record producer.
Disable Direct	disable_direct	bool	false	Disable the in-process listener-to-target fast path. When false, the direct path engages automatically when eligible.

Collector Resource Management

Controls resource allocation and scheduling for device collectors.

Field	YAML Path	Type	Default	Description
Retry Interval	retry_interval	int	30	Device retry delay in seconds
Start Delay	start_delay	int	1	Collector startup delay in seconds
Windows Start	windows_start	int	524288000 (500 MB)	Memory threshold for Windows collectors (bytes)
Windows Start Buffer	windows_start_buffer	bool	--	Buffer mode for Windows collectors
Monitor Interval	monitor_interval	int	30	Monitor interval in seconds
Flush Interval	flush_interval	int	1	Log flush interval in seconds
Modules	modules	bool	true	Enable or disable module loading

Profiler

Field	YAML Path	Type	Default	Description
Profile Port	profiler.director_profile_port	int	6061	pprof profiler port
Timeout	profiler.timeout	int	60	Profiler and metrics collection timeout in seconds

Load Balancer

Field	YAML Path	Type	Default	Description
Mode	loadbalancer.mode	string	--	Set to dynamic for dynamic load balancing across cluster nodes

Persistent Storage

Controls whether pipeline data survives a restart, and how far upstream that durability extends. The two flags combine into four modes:

persistent_storage.status	crash_resistance	Effect
false (default, standalone)	forced false	In-memory VMFL/VMF and JetStream MemoryStorage for the pipeline streams. Maximum throughput; records buffered in process memory are lost on a crash.
true	false	JetStream FileStorage plus the NATS Object Store for payloads. VMFL/VMF writers stay in-memory; durability is provided by NATS.
true	true	Enterprise: local VMFL/VMF queue writers also go to disk, flushing each record to the OS page cache -- process-crash durable end-to-end.
false	true	Ignored -- crash_resistance is forced false and the mode collapses to the first row. The override is silent: no warning is logged.

note

The first row applies to the three pipeline streams (ingester, router, sender). The two fleet control streams and the Director data stream are created with FileStorage regardless of persistent_storage.status, so some disk is reserved in every mode.

Field	YAML Path	Type	Default	Description
Status	persistent_storage.status	bool	false	Enable the durable NATS JetStream pipeline layer (JetStream FileStorage + NATS Object Store). Defaults to true on fleet-managed Directors.
Crash Resistance	persistent_storage.crash_resistance	bool	false	Also write the local VMFL/VMF queue files to disk, flushing each record to the OS page cache so the upstream pipeline hops survive a process crash (SIGKILL, OOM, panic). Enterprise deployments; only effective when persistent_storage.status: true (forced false otherwise).
Max Age	persistent_storage.max_age	numeric	172800	Retention horizon in seconds for the durable streams (48 hours). A non-positive value falls back to the default.

note

crash_resistance targets process-crash durability, not power loss. With persistent_storage.status: true, power-loss durability is provided by NATS -- best-effort on a single node (default 2-minute sync interval) or per-write via RAFT quorum on a cluster. For power-loss-sensitive workloads, run a cluster.

In clustered or environment-file deployments, place the persistent_storage block -- including its nested payload and pipeline_bus -- under a node's or the cluster's properties: block rather than at the top level of vmetric.yml (see the persistent-storage configurations under Examples).

Cluster properties are applied first and node properties are applied on top, so a value set on a node overrides the cluster value for that node. Nothing validates consistency across a cluster: divergent values are accepted silently and each node runs with its own resolved flags. Keep them uniform -- nodes of a cluster share the same NATS and pipeline layer, and mixed durability settings produce behavior that is difficult to reason about.

For Orchestrated Directors, the processing mode and the external storage connection are also configurable through the web interface -- see Persistent Storage.

Payload

Configures the payload backend that carries log record bytes between pipeline stages. The payload and pipeline_bus blocks nest inside persistent_storage; the full key paths are persistent_storage.payload.* and persistent_storage.pipeline_bus.*.

Field	YAML Path	Type	Default	Description
Type	persistent_storage.payload.type	string	""	Payload backend: inline (bytes carried in the pipeline message), object_store (NATS Object Store), memory (in-process store), or "" (automatic — memory when persistent storage is off, object_store when on).
Max Size	persistent_storage.payload.max_size	string	""	Maximum payload size per record. Accepts a plain integer (bytes) or a size suffix such as 1mb or 512kb. When the payload type is inline defaults to 1 MB to fit within Kafka and Event Hubs message limits; unlimited for other backends.

Pipeline Bus

Configures the transport that moves stage-to-stage pipeline notifications between the ingester, router, and sender workers.

Field	YAML Path	Type	Default	Description
Type	persistent_storage.pipeline_bus.type	string	nats	Stage-to-stage transport: nats (embedded VMMQ, default), kafka (external Kafka broker), or eventhubs (Azure Event Hubs, treated as kafka against its Kafka-compatible endpoint). An unrecognized value logs a warning and falls back to nats.
Address	persistent_storage.pipeline_bus.address	string	--	Comma-separated Kafka broker addresses. Required when the pipeline bus type is kafka.
Topic Prefix	persistent_storage.pipeline_bus.topic_prefix	string	vm-pipeline	Prefix for stage topics. Topics are named <prefix>.<stage> (e.g., vm-pipeline.sender).
Algorithm	persistent_storage.pipeline_bus.algorithm	string	--	Kafka SASL algorithm: plain, scram-sha-256, scram-sha-512, or gssapi.
Username	persistent_storage.pipeline_bus.username	string	--	Kafka SASL username.
Password	persistent_storage.pipeline_bus.password	string	--	Kafka SASL password.
TLS Status	persistent_storage.pipeline_bus.tls.status	bool	false	Enable TLS for the Kafka broker connection.
TLS Insecure Skip Verify	persistent_storage.pipeline_bus.tls.insecure_skip_verify	bool	false	Skip broker certificate verification.
TLS Min Version	persistent_storage.pipeline_bus.tls.min_tls_version	string	tls1.2	Minimum TLS version for the broker connection.
TLS Max Version	persistent_storage.pipeline_bus.tls.max_tls_version	string	tls1.3	Maximum TLS version for the broker connection.

note

pipeline_bus.type: kafka requires payload.type: inline. If payload.type is not inline, the Director logs a warning and reverts to the nats transport.

---

Environment Variables

Variable	Value	Effect
virtualmetric	debug	Loads vmetric-local.yml instead of vmetric.yml

Debug config file lookup order:

1. {serviceName}-local.toml
2. {serviceName}-local.yml
3. {serviceName}-local.yaml