Skip to main content

Targets: Overview

Targets are the final stage in the DataStream processing flow. They forward processed data to external consumers and convert from standardized pipeline output to destination-specific formats.

Provider → Device → Preprocessing → Pipeline → Postprocessing → Target → Consumer

Image targets-flow

Targets provide flexible options for data persistence and integration with various analysis platforms.

Definitions​

Targets serve as the endpoint destinations for processed data within the Director system. They operate on the following principles:

  1. Output Configuration: Targets define where and how data should be delivered after processing.
  2. Format Adaptation: They handle the conversion of internal data structures to destination-specific formats.
  3. Delivery Management: Targets manage connection pooling, batching, and retry mechanisms.
  4. Destination Integration: They provide authentication and protocol-specific features for various systems.
note

Targets enable:

Persistence: Local and remote storage with various retention options.
Integration: Seamless connection to analytics platforms and messaging systems.

They also support data transformation, and delivery confirmation.

Target Architecture​

DataStream uses a small number of target implementations that each support multiple integrations (services). This efficiency pattern means similar services share implementation code:

ImplementationIntegrations Supported
awss3TargetAmazon S3, Security Lake, Cloudflare R2, MinIO, DigitalOcean Spaces, and 6 others
kafkaTargetApache Kafka, Confluent Cloud, Aiven Kafka, Redpanda, WarpStream, Amazon MSK, IBM Event Streams
elasticTargetElasticsearch, Amazon OpenSearch, Elastic Security
splunkTargetSplunk, Splunk Security
syslogSIEMTargetArcSight, OpenText, Trellix, Snare, Logpoint (CEF/JSON over syslog)
gelfTargetGraylog, OVHcloud (GELF)
siemHTTPTargetDatadog, Sumo Logic, Rapid7 (JSON over HTTPS)

Each integration gets its own documentation chapter with platform-specific details, even when the underlying implementation is shared.

Postprocessing Pipelines​

Targets can have pipelines attached for output format transformation via the pipelines field. These postprocessing pipelines convert standardized pipeline output to target-specific formats before delivery.

Common postprocessing tasks include:

  • Field mapping to destination schemas (ECS, CIM, ASIM)
  • Format conversion for target requirements
  • Data filtering before delivery

Configuration​

All targets share the following base configuration fields:

FieldRequiredDefaultDescription
nameYUnique identifier for the target
descriptionNOptional explanation
typeYTarget type
statusNtrueEnable/disable the target
batch_sizeN1000Records per batch
tip

Each target type provides specific configuration options detailed in their respective sections.

Use the name of the target to refer to it in your configurations.

Example:

targets:
  - name: elasticsearch
    type: elasticsearch
    properties:
      hosts: ["http://elasticsearch:9200"]
      index: "logs-%{+yyyy.MM.dd}"
      username: "elastic"
      password: "${PASSWORD}"

The target listed here is of type elasticsearch. It specifies the host that it will forward the data to and the index on the host to which the data will be appended. To access the host, it specifies a username and a password.

tip

You can use environment variables like ${PASSWORD} for your credentials. This will improve security by removing the credentials from your configuration file.

Target Types​

The system supports the following target types, grouped as they appear in the sidebar:

  • Microsoft Azure — Azure service integrations:

    • Microsoft Sentinel: Ingests into Sentinel workspace tables using Data Collection Rules, with automatic table selection and field normalization
    • Microsoft Sentinel data lake: Ingests into data lake tables for lower-cost, extended-retention storage of high-volume log types
    • Azure Blob Storage: Writes files to blob containers with configurable formats, authentication methods, and rotation
    • Azure Data Explorer: Ingests directly into Kusto tables, supporting multiple tables and custom schemas
    • Azure Event Hubs: Streams events to Event Hubs with batching and several authentication methods
    • Azure Monitor Logs: Ingests into Log Analytics workspace tables using Data Collection Rules, with Managed Identity support
    • Azure Service Bus: Sends messages to queues or topics with batching and advanced message properties

  • Amazon Web Services (AWS) — AWS service integrations:

    • Amazon Security Lake: Writes OCSF-formatted security data to Security Lake
    • Amazon S3: Writes files to S3 buckets with multipart uploads and size-based rotation
    • Amazon CloudWatch Logs: Writes log events to CloudWatch Logs with configurable batch limits
    • Amazon Kinesis: Writes records to Kinesis Data Streams with batching
    • Amazon MSK: Writes to managed Kafka topics with batching, compression, and AWS authentication
    • Amazon OpenSearch: Sends via the Bulk API with IAM authentication and load balancing across nodes
    • Amazon Redshift: Loads data through S3 staging and the COPY command
    • Amazon SNS: Publishes messages to SNS topics with batching
    • Amazon SQS: Sends messages to SQS queues with batching

  • Google Cloud (GCP) — Google Cloud service integrations:

    • Google SecOps: Forwards to Google Security Operations via the Ingestion API, as unstructured logs or UDM
    • Google Chronicle: Sends to Chronicle via the Ingestion API with OAuth2 service-account authentication
    • Google BigQuery: Streams rows into BigQuery tables using the streaming insert API
    • Google Cloud Logging: Forwards entries with configurable severity levels and labels
    • Google Cloud Pub/Sub: Publishes to Pub/Sub topics with batching and message ordering
    • Google Cloud Storage: Writes files to GCS buckets with multipart uploads and rotation

  • IBM Cloud — IBM Cloud service integrations:

    • IBM Cloud Object Storage: Writes files to COS buckets with configurable formats and rotation
    • IBM Event Streams: Writes to IBM's managed Kafka service with full Kafka API compatibility
    • IBM Cloud Logs: Forwards events via the Logs API singles endpoint, with application and subsystem context

  • Message Queues — Messaging platform publishers:

    • Apache Kafka: Writes to Kafka topics with batching, compression, and several authentication methods
    • Confluent Cloud: Writes to Confluent Cloud topics
    • Synadia Cloud: Publishes to Synadia's managed NATS JetStream service
    • Redpanda: Writes to Redpanda topics
    • WarpStream: Writes to WarpStream's serverless, Kafka-compatible platform
    • Aiven Kafka: Writes to Aiven for Apache Kafka, which requires TLS and SASL
    • NATS: Publishes to NATS JetStream subjects
    • RabbitMQ: Publishes to exchanges or queues with routing
    • MQTT: Publishes to MQTT topics with configurable QoS levels

  • Analytics — Analytics platform integrations. The Databricks and Snowflake targets are staging-based: they write files to object storage first, then issue a load command.

    • ClickHouse: Sends to a ClickHouse server with batching and field normalization
    • Elasticsearch: Sends via the Bulk API with load balancing across nodes
    • Elastic Security: Sends via the Bulk API with security index patterns and ECS normalization
    • Splunk: Sends via the HTTP Event Collector with load balancing across endpoints
    • Splunk Enterprise Security: Sends via HEC with security index configuration and CIM normalization
    • Databricks (S3 Staging): Stages files to Amazon S3, then loads into Unity Catalog tables with COPY INTO
    • Databricks (Azure Blob Storage): Stages files to Azure Blob Storage, then loads into Unity Catalog tables with COPY INTO
    • Snowflake (S3 Staging): Stages files to Amazon S3, then loads into Snowflake tables with COPY INTO
    • Snowflake (Azure Blob Staging): Stages files to Azure Blob Storage, then loads into Snowflake tables with COPY INTO

  • SIEM — SIEM platform integrations. Most are transport-only: the event is expected to already be in the platform's wire format when it reaches the target, which handles framing and delivery. Format conversion is a pipeline concern.

    • IBM QRadar: Forwards LEEF-formatted events to an Event Collector over syslog
    • ArcSight: Forwards CEF-formatted events over syslog
    • OpenText: Forwards CEF-formatted events to OpenText Security Log Analytics over syslog
    • Trellix: Forwards CEF-formatted events to Trellix SIEM / McAfee ESM over syslog
    • Snare: Forwards CEF-formatted events to Snare (Prophecy) over syslog
    • Logpoint: Forwards JSON-formatted events over syslog
    • Graylog: Forwards GELF-formatted events to a GELF input over TCP, TLS, UDP, or HTTP(S)
    • OVHcloud Logs Data Platform: Forwards GELF-formatted events to the OVHcloud GELF input
    • Datadog: Forwards JSON events to the Cloud SIEM logs intake API over HTTPS
    • Sumo Logic: Forwards JSON events to a Cloud SIEM HTTP Source collector
    • Rapid7 InsightIDR: Forwards JSON events to a log HTTP ingestion endpoint
    • CrowdStrike Falcon Next-Gen SIEM: Sends events natively over its HTTP Event Collector endpoint

  • Cloud Storage — S3-compatible object storage. All share the Amazon S3 implementation, so they take the same file formats, rotation, and multipart upload behavior, differing only in endpoint and credentials.

    • MinIO: Writes to MinIO object storage
    • Cloudflare R2: Writes to R2 buckets
    • DigitalOcean Spaces: Writes to Spaces buckets
    • Scaleway Object Storage: Writes to Scaleway buckets
    • Oracle Cloud Object Storage: Writes to OCI Object Storage buckets
    • Alibaba Cloud OSS: Writes to Alibaba Object Storage Service buckets
    • Backblaze B2 Cloud Storage: Writes to B2 buckets
    • Wasabi Hot Cloud Storage: Writes to Wasabi buckets

  • Other — Local output and generic protocols:

    • Console: Writes to standard output, with field normalization
    • Discard: Drops all incoming messages
    • File: Writes local files as JSON, MultiJSON, Avro, or Parquet, with compression and rotation
    • HTTP: Sends to HTTP/HTTPS endpoints with configurable methods, formats, and authentication
    • OTLP: Forwards to any OpenTelemetry-compatible receiver over HTTP or gRPC
    • Syslog: Forwards to a remote syslog server in RFC5424 format, with templating and TLS
    • TCP: Forwards to a remote TCP endpoint with delimiter or octet-count framing
    • UDP: Sends each event as an individual datagram, without batching
note

Several of these integrations share a single implementation — the eight Cloud Storage targets and Amazon S3 are one code path, as are the Kafka-family targets. Configuration is therefore near-identical within those families; the per-target pages document the endpoint and authentication differences that matter.

Scheduling and Pool Behavior​

Every target runs in a dedicated worker pool. Scheduling and parallelism fields apply uniformly to every target type and live directly inside the target's properties: block:

targets:
  - name: my_target
    type: <any-target-type>
    properties:
      # ... target-specific fields ...
      interval: 300                 # flush every 5 minutes
      cron: "0 20 * * *"            # OR flush nightly at 20:00 (cron wins if both set)
      queue:
        parallelism: 1              # worker count override (optional)
FieldRequiredDefaultDescription
intervalN-Flush every N seconds (fixed cadence)
cronN-Flush per cron expression (e.g. 0 20 * * * = every day at 20:00)
queue.parallelismNservice workers valueNumber of pinned worker goroutines for this target
queue.intervalN-Legacy fallback location for interval (nested under queue:)
queue.cronN-Legacy fallback location for cron (nested under queue:)

Mode selection​

interval set?cron set?Resulting mode
NoNoImmediate — flush once per incoming notification
YesNoScheduled (interval) — accumulate, flush every N seconds
NoYesScheduled (cron) — accumulate, flush per cron expression
YesYesScheduled (cron) — cron wins; interval is ignored

In scheduled mode the pool calls the target's Send for every event but defers Finalize until the schedule fires. This produces "one batch per tick" semantics — important for file-like targets (file, awss3, azblob, gcpstorage, gcpsecops) where each tick produces a single output file.

Parallelism​

File-like targets force queue.parallelism: 1 regardless of the configured value — concurrent writers cannot produce a single coherent file per tick. Network and analytics targets default to the service-level workers setting (minimum 2 when unspecified).

Bad cron expressions​

A cron value that cannot be parsed causes the target pool to fail at startup. Validate cron syntax with a fixture run before promoting to production.

Rate Limiting​

Optional per-target throttling caps how fast and how much data a target accepts. Configure it with a limit block. Both caps are disabled by default, and throttling is independent of batch_size and worker parallelism.

FieldTypeDefaultDescription
epsnumeric0 (unlimited)Maximum events per second.
daily_gbnumeric0 (unlimited)Maximum data volume per day, in GiB. The counter resets at midnight (local time).
behaviourstringdelayAction when a cap is reached: delay or drop.

behaviour controls how overflow is handled:

  • delay — events are paced to stay within eps; once daily_gb is reached, delivery is held and retried until the daily counter resets at midnight. No data is discarded.
  • drop — events beyond eps (within a one-second window) or arriving after daily_gb is reached are discarded silently.

An unknown or empty behaviour falls back to delay, so a misconfiguration preserves data rather than dropping it.

targets:
  - name: my_target
    type: ...
    limit:
      eps: 5000
      daily_gb: 50
      behaviour: delay
note

Editing a target's limit recreates its worker pool, which resets the daily byte counter. A negative daily_gb is rejected and the target will not start.

Targets support debug configuration options for testing, troubleshooting, and development purposes. These options allow you to inspect data flow without affecting production systems.

Configuration​

Debug options are configured under the debug property within target properties:

targets:
  - name: test_elastic
    type: elastic
    properties:
      index: "test-logs"
      endpoints:
        - endpoint: "http://elasticsearch:9200"
      debug:
        status: true
        dont_send_logs: false

Debug Fields​

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging for the target
debug.dont_send_logsNfalsePrevent logs from being sent to the actual target

Debug Status​

When debug.status is set to true, the target logs each event to the internal debugger before processing. This provides visibility into:

  • Message content being sent
  • Device information (ID, name, type)
  • Target type and operation details
  • Timing and sequence of events

Debug logs are written to the system's debug output and can be used to:

  • Verify data transformation and formatting
  • Troubleshoot pipeline processing issues
  • Monitor data flow in development environments
  • Audit message content during testing

Don't Send Logs​

When debug.dont_send_logs is set to true, events are logged to the debugger but not sent to the actual target destination. This is useful for:

  • Safe Testing: Test configuration changes without affecting production systems
  • Development: Develop and validate pipelines without external dependencies
  • Cost Control: Avoid charges from cloud services during testing
  • Dry Runs: Verify event formatting and routing logic before deployment
warning

The dont_send_logs option only works when debug.status is also set to true. If debugging is disabled, logs will be sent normally regardless of the dont_send_logs setting.

Use Cases​

Development Environment​

Test your configuration safely without sending data to production targets:

targets:
  - name: dev_splunk
    type: splunk
    properties:
      endpoints:
        - endpoint: "https://splunk.example.com:8088/services/collector"
          token: "YOUR-TOKEN"
      index: "main"
      debug:
        status: true
        dont_send_logs: true

Troubleshooting​

Enable debug logging to diagnose issues while still sending data:

targets:
  - name: debug_elastic
    type: elastic
    properties:
      index: "production-logs"
      endpoints:
        - endpoint: "http://elasticsearch:9200"
      debug:
        status: true
        dont_send_logs: false

Pipeline Validation​

Verify pipeline transformations before enabling the target:

targets:
  - name: validate_transformations
    type: splunk
    properties:
      endpoints:
        - endpoint: "https://splunk.example.com:8088/services/collector"
          token: "YOUR-TOKEN"
      field_format: "cim"
      debug:
        status: true
        dont_send_logs: true
    pipelines:
      - name: test_pipeline
        processors:
          - set:
              field: environment
              value: "development"

Staged Deployment​

Test new target configurations in parallel with existing ones:

targets:
  # Production target (normal operation)
  - name: prod_elastic
    type: elastic
    properties:
      index: "production-logs"
      endpoints:
        - endpoint: "http://prod-elasticsearch:9200"
  
  # Test target (debug mode, no actual sending)
  - name: test_elastic
    type: elastic
    properties:
      index: "test-logs"
      endpoints:
        - endpoint: "http://test-elasticsearch:9200"
      debug:
        status: true
        dont_send_logs: true

Deployment​

The following deployment types can be used:

  • One-to-many - data from a single source is routed to one or more destinations:

    Syslog → Local Storage + Analysis Platform

  • Many-to-one - data from multiple sources is routed to one destination:

    Syslog + Windows → Local Storage

  • Many-to-many - data from multiple sources is routed to multiple destinations:

    Syslog + Windows → Local Storage

    Syslog + Elasticsearch → Cloud Upload + Analysis Platform

  • Chained - data is routed sequentially from one destination to the next:

    Syslog → Local Storage → Analysis Platform

Multiple targets can be used for redundancy, normalization rules can be implemented, and alerts can be put in place for notification and error handling.

Use Cases​

The most common uses of targets are:

  • Local analysis - Debug logging, performance analysis, audit trails, and temporary storage.

  • Cloud integration - Long-term storage, data warehousing, security analysis, and compliance monitoring.

  • Real-time analysis - Live monitoring, alert generation, trend analysis, and performance tracking.

  • Data lake building - Raw data storage, schema evolution, data partitioning, and analytics preparation.

To serve these ends, the following processing options are available:

  • Pipelines - Field normalization (for ECS, CIM, ASIM, CEF, LEEF, and CSL), data transformation, message batching, custom field mapping, schema validation, and format conversion.

  • Buffer management - Configurable buffer sizes, batch processing, flush intervals, queue management, checkpoint recovery, and error handling.

  • Performance - Asynchronous writing, buffer optimization, connection pooling, retry mechanisms, resource monitoring, and size-based rotation.

  • Security - Authentication using API keys, service principals, and client certificates. Encryption with TLS/SSL, HTTPS, or custom algorithms. Also, access control and audit logging.

Implementation Strategies​

The following strategies configure target output and storage options.

Output Form Factors​

Independently of which service a target writes to, its output takes one of three shapes. This determines what you configure for buffering, rotation, and delivery. See Target Types above for the full list of destinations.

  • Streamed — each event, or a batch of events, is sent over a live connection as it arrives. Network and messaging targets work this way. Configuration centers on batching, connection reuse, and retries.

  • File-based — events accumulate into files that are closed and shipped on a size or time boundary. Local files and every object-storage target work this way. Configuration centers on format, compression, and rotation.

  • Staged — files are written to object storage first, then a load command ingests them into the destination. The Redshift, Databricks, and Snowflake targets work this way, so they need both the staging bucket's configuration and the warehouse's.

File Formats​

The file-based and staged form factors write one of the following:

  • json — each log entry is written as a separate JSON line (JSONL format)
  • multijson — all log entries are written as a single JSON array
  • avro — Apache Avro format with schema
  • parquet — Apache Parquet columnar format with schema

Compression options like ZSTD, GZIP, Snappy, Brotli, and LZ4 are also supported. Additional features include dynamic file naming, size-based rotation, buffer management, and schema validation.

  • Microsoft Sentinel - Direct DCR integration and ASIM normalization are supported. In addition to standard tables, WindowsEvent, SecurityEvent, CommonSecurityLog, and Syslog can be used. Various ASIM tables are also available. (See the ASIM section for a complete list.)