Skip to main content
Version: 1.6.1

Introduction

Directors are the core data processing engines within the DataStream platform, responsible for collecting, processing, transforming, and routing security telemetry data from various sources to target destinations. They serve as the central orchestration layer that maintains data sovereignty by keeping sensitive information within your environment while providing centralized cloud-based management.

What is a Director?​

A Director is a lightweight, containerized service that acts as a secure data processing hub in your infrastructure. It connects securely to the DataStream cloud platform for configuration management while ensuring all sensitive security data remains within your controlled environment.

Key Capabilities​

A Director offers the following capabilities:

Data Processing Pipeline:

  • Ingests security data from multiple sources (syslog, APIs, files, databases)
  • Applies real-time transformation and normalization using YAML-defined pipelines
  • Supports multiple security schemas (ASIM, OCSF, ECS, CIM, UDM)
  • Routes processed data to various destinations (SIEM platforms, data lakes, security tools)

Security and Compliance:

  • Maintains data sovereignty by processing all data locally
  • Establishes outbound-only HTTPS connections to cloud management services
  • Provides comprehensive audit logging and activity tracking
  • Supports enterprise security requirements and compliance frameworks

Scalability and Reliability:

  • Horizontal scaling through clustering capabilities
  • High availability configurations for mission-critical environments
  • Resource-efficient processing with minimal infrastructure requirements
  • Automatic failover and load balancing in clustered deployments

Platform Management Options​

DataStream provides two distinct management approaches for Directors, each designed for different organizational needs and security requirements:

Self-Managed Directors​

Self-Managed Directors provide complete control over the deployment and management of your data processing infrastructure. This option is ideal for organizations with specific security requirements or existing infrastructure management processes.

Characteristics:

  • Full control over deployment environment and configuration
  • Direct management of updates, patches, and maintenance
  • Custom security controls and compliance configurations
  • Integration with existing infrastructure monitoring and management tools
  • Support for air-gapped or restricted network environments

Suitable For:

  • Organizations with strict data governance requirements
  • Environments with existing container orchestration systems
  • Companies requiring custom security configurations
  • Regulated industries with specific compliance needs

Managed Directors (Enterprise Feature)​

Managed Directors offer a fully-managed service where VirtualMetric handles the infrastructure management, monitoring, and maintenance of your Directors while still maintaining data sovereignty.

Characteristics:

  • Automated deployment and configuration management
  • Proactive monitoring and maintenance by VirtualMetric
  • Automatic updates and security patches
  • 24/7 support and incident response
  • Performance optimization and capacity planning

Suitable For:

  • Organizations seeking reduced operational overhead
  • Teams without dedicated infrastructure management resources
  • Companies prioritizing time-to-value over operational control
  • Environments requiring guaranteed SLA and support coverage

Installation Types​

Directors support different installation architectures to accommodate various operational requirements and scale needs:

Standalone Installation​

Standalone is the default installation type, designed for straightforward deployments where a single Director instance handles all data processing needs.

Features:

  • Single Director instance per deployment
  • Simplified configuration and management
  • Resource-efficient for most use cases
  • Quick deployment and setup process

Limitations:

  • No built-in high availability or load balancing
  • Single point of failure for data processing
  • Limited horizontal scaling capabilities
  • Manual backup and disaster recovery procedures

Recommended For:

  • Small to medium-scale deployments
  • Development and testing environments
  • Organizations with basic availability requirements
  • Initial proof-of-concept implementations

Clustered Installation (Enterprise Feature)​

Clustered installations provide high availability and horizontal scaling capabilities through multiple Director instances working together.

Features:

  • Multiple Director instances with automatic load balancing
  • Built-in failover and redundancy mechanisms
  • Horizontal scaling based on processing demands
  • Distributed processing for improved performance
  • Shared state management across cluster nodes

Benefits:

  • Elimination of single points of failure
  • Improved processing capacity and throughput
  • Automatic recovery from node failures
  • Dynamic scaling based on data volume
  • Enhanced monitoring and observability

Recommended For:

  • Mission-critical security data processing
  • High-volume environments requiring guaranteed availability
  • Organizations with strict SLA requirements
  • Production deployments requiring enterprise-grade reliability

Director Architecture and Data Flow​

Directors operate as secure intermediaries between your security data sources and target destinations, implementing a data sovereignty model that keeps sensitive information within your controlled environment.

Data Processing Architecture​

Input Layer:

  • Multiple simultaneous data source connections
  • Protocol-agnostic ingestion (Syslog, REST APIs, file monitoring)
  • Real-time streaming and batch processing capabilities
  • Built-in buffering and queuing for reliability

Processing Layer:

  • YAML-defined transformation pipelines
  • Multi-schema normalization and enrichment
  • Real-time data validation and quality checks
  • Custom logic implementation through processors

Output Layer:

  • Multi-destination routing and delivery
  • Format adaptation for different target systems
  • Delivery confirmation and retry mechanisms
  • Performance optimization for various endpoint types

Security and Connectivity Model​

Outbound-Only Communication:

  • Directors initiate all cloud platform connections
  • No inbound firewall rules required
  • Encrypted HTTPS communication for all cloud interactions
  • Certificate-based authentication and authorization

Data Sovereignty:

  • All security data processing occurs locally
  • No sensitive data transmitted to cloud services
  • Configuration and metadata-only cloud synchronization
  • Complete audit trail for compliance and governance