Elastic Security

SIEM Platform Observability

Synopsis

The Elastic Security target sends security telemetry to Elastic Security (formerly Elastic SIEM) using the Elasticsearch Bulk API with security-optimized index patterns and ECS normalization. This target extends standard Elasticsearch configuration with security-specific index management and detection rule integration.

Schema

- name: <string>
  description: <string>
  type: elasticsecurity
  properties:
    version: <integer>
    index: <string>
    endpoints:
      - endpoint: <string>
        username: <string>
        password: <string>
    max_payload_size_kb: <integer>
    batch_size: <integer>
    timeout: <integer>
    use_compression: <boolean>
    write_action: <string>
    field_format: <string>
    pipeline: <string>
    filter_path: <string>
    insecure_skip_verify: <boolean>
    ca_name: <string>
    server_name: <string>
    cert_name: <string>
    key_name: <string>
    passphrase: <string>
    min_tls_version: <string>
    max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be elasticsecurity
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Elastic Security Connection

Field	Required	Default	Description
version	N	auto	Accepted but not applied. The value is stored and never read, so no version detection or version-specific behavior follows from it
index	Y	-	Default security index name (e.g., logs-security.events-default)
endpoints	Y	-	Array of Elasticsearch endpoint configurations
endpoints[].endpoint	Y	-	Elasticsearch URL (e.g., https://elastic.example.com:9200)
endpoints[].username	N	-	Basic authentication username
endpoints[].password	N	-	Basic authentication password

Bulk API Configuration

Field	Required	Default	Description
max_payload_size_kb	N	4096	Maximum bulk request size in KB
batch_size	N	10000	Maximum number of events per batch
timeout	N	30	Connection timeout in seconds
use_compression	N	true	Enable GZIP compression
write_action	N	create	Bulk API action (index, create)

Security Platform Configuration

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section
pipeline	N	-	Elasticsearch ingest pipeline name
filter_path	N	errors,items.*.error,items.*._index,items.*.status	Response filter path

TLS

TLS is engaged by the endpoint URL scheme: an https:// endpoint enables it, an http:// endpoint does not. There is no status field. The fields below are top-level, at the root of properties.

The web interface exposes only insecure_skip_verify for these targets. Every other field below — the CA bundle, the client certificate and key, the SNI override and the version bounds — has to be set in the YAML configuration or through the API; there is no form control for it.

Field	Required	Default	Description
insecure_skip_verify	N	false	Skip server certificate verification. Use only for testing.
ca_name	N	-	CA bundle used to verify the server certificate. When unset, the host trust store is used; when set, it replaces the host trust store rather than adding to it.
server_name	N	-	SNI hostname override for the TLS handshake
cert_name	N*	-	Client certificate for mutual TLS
key_name	N*	-	Client private key for mutual TLS
passphrase	N	-	Passphrase for an encrypted private key
min_tls_version	N	tls1.2	Minimum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3)
max_tls_version	N	tls1.3	Maximum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3)

* = Mutual TLS requires both cert_name and key_name. Setting only one is a configuration error.

note

A nested tls: block is also accepted, and a non-empty one replaces the flat shape rather than merging with it: the sender reads the block and ignores every flat key, including options the block leaves out. Use one shape or the other, never a mixture.

insecure_skip_verify is the single exception. It is read from the nested block when present and otherwise falls back to the root, so a target carrying a hand-written tls: block keeps the skip-verify its form wrote. Only an absent nested value falls through — a nested false is an explicit statement and overrides a root true.

There is no tls.status: TLS is off when the whole block is absent, and a status key inside the block is not read.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Elastic Security

Details

Security Index Patterns

Recommended Index Patterns:

• logs-security.events-*: Security events and alerts
• logs-security.auth-*: Authentication and authorization logs
• logs-security.network-*: Network traffic and firewall logs
• logs-security.endpoint-*: Endpoint security events
• logs-security.threat-*: Threat intelligence indicators

Index Naming Convention:

• Format: logs-{dataset}-{namespace}
• Dataset: Security data category
• Namespace: Environment or team identifier (e.g., default, production)

Data Stream Compatibility

Use index patterns compatible with Elastic data streams for automatic index lifecycle management and rollover.

ECS Normalization

Elastic Common Schema (ECS):

• Apply field_format: ecs for automatic field normalization
• Ensures compatibility with Elastic Security detection rules
• Provides consistent field naming across data sources
• Enables the pre-built Elastic Security views and visualizations

Critical ECS Fields:

• @timestamp: Event timestamp (ISO 8601)
• event.category: Event categorization (authentication, network, process, file)
• event.type: Event type (start, end, info, error)
• source.ip / destination.ip: Network addresses
• user.name: Username for authentication events
• process.name: Process information for endpoint events

Detection Rule Integration

Rule Compatibility:

• Elastic Security detection rules require ECS-normalized data
• Custom rules can reference specific index patterns
• Pre-built rules work automatically with standard ECS fields

Index Configuration:

• Configure indices in Elastic Security � Management � Index Patterns
• Create index patterns matching your configured target indices
• Enable index patterns in Detection Rules

Performance Optimization

Batch Configuration:

• Larger batches reduce API calls and improve throughput
• Balance batch size against memory usage and latency
• Default batch_size: 10000 suitable for most workloads

Compression:

• GZIP compression enabled by default
• Reduces network bandwidth by 70-90%
• Minimal CPU overhead for compression/decompression

Load Balancing:

• Configure multiple endpoints for high availability
• Events distributed randomly across endpoints
• Automatic failover to available endpoints

Index Mapping

Ensure Elasticsearch index mappings match ECS field types. Incorrect mappings cause ingestion failures. Use index templates for automatic mapping configuration.

Examples

Basic Security Events

Sending security events to Elastic Security with ECS normalization...

targets: - name: elastic-security-events type: elasticsecurity properties: index: logs-security.events-default field_format: ecs endpoints: - endpoint: https://elastic.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

Authentication Logs

Forwarding authentication logs to dedicated security index...

targets: - name: elastic-auth-logs type: elasticsecurity properties: index: logs-security.auth-production field_format: ecs use_compression: true endpoints: - endpoint: https://elastic.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

High-Availability Configuration

Multiple Elasticsearch endpoints for high availability and load balancing...

targets: - name: elastic-security-ha type: elasticsecurity properties: index: logs-security.events-default field_format: ecs batch_size: 10000 max_payload_size_kb: 4096 use_compression: true endpoints: - endpoint: https://elastic-node1.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}" - endpoint: https://elastic-node2.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}" - endpoint: https://elastic-node3.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

With Ingest Pipeline

Using Elasticsearch ingest pipeline for additional processing...

targets: - name: elastic-security-pipeline type: elasticsecurity properties: index: logs-security.network-default field_format: ecs pipeline: security-enrichment endpoints: - endpoint: https://elastic.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

Threat Intelligence

Forwarding threat intelligence indicators to Elastic Security...

targets: - name: elastic-threat-intel type: elasticsecurity properties: index: logs-security.threat-default field_format: ecs write_action: create endpoints: - endpoint: https://elastic.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

Endpoint Security Events

Sending endpoint detection events to Elastic Security...

targets: - name: elastic-endpoint-security type: elasticsecurity properties: index: logs-security.endpoint-default field_format: ecs batch_size: 5000 use_compression: true endpoints: - endpoint: https://elastic.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

Production Configuration

Production-ready Elastic Security configuration with HA, compression, and ECS normalization...

targets: - name: elastic-security-production type: elasticsecurity properties: index: logs-security.events-production field_format: ecs batch_size: 10000 max_payload_size_kb: 4096 timeout: 30 use_compression: true write_action: create insecure_skip_verify: false endpoints: - endpoint: https://elastic-node1.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}" - endpoint: https://elastic-node2.example.com:9200 username: "${ELASTIC_USERNAME}" password: "${ELASTIC_PASSWORD}"

Troubleshooting

The elasticsecurity target behaves exactly the same way as the elastic target. It builds the same bulk requests, authenticates the same way, reports the same errors for the same causes, and retries or drops a payload on the same rules. There is no behavioral difference between the two. Use the Elasticsearch troubleshooting section for the full list of errors, causes and fixes: credentials rejected, documents the index refuses, size limits, throttling, transport and certificate failures, and the cases where the target is connected but nothing arrives.

See Target Delivery Errors for how Director logs and retries target failures.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". Everything up to from Elastic: is written by Director, and what follows it is the cluster's own answer.
• The target's connection status in the web interface. It shows the same reason as the log line.

Four things to keep in mind while reading that section for this target:

• Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples.
• The security index names on this page follow the logs-{dataset}-{namespace} convention, and names in that shape are usually data streams. A data stream accepts only the create action, so keep the default write_action: create. With write_action: index the cluster refuses every document, typically with an illegal_argument_exception naming the op type, reported one segment per document, and the whole payload is dropped after four deliveries.
• The index privilege described in the 403 entry has to be granted on every pattern this target writes to, logs-security.events-*, logs-security.auth-* and any other name a pipeline puts in the index field, not only on the value of index.
• field_format: ecs marks every record as a JSON document, so the failed to parse JSON message entry applies to all of them rather than to the occasional record. A record that is not a JSON object is never accepted and never dropped, and it holds up everything queued behind it, so treat that error as urgent here.

tip

The detection rules that read these indices depend on ECS field types. A field that arrives with the wrong type is refused per document, which is the bulk operation had errors entry, and that entry is the one where data is lost. Fix the mapping or normalize the field in a Director pipeline before the fourth delivery.