Elastic

Synopsis

Creates an HTTP listener that emulates the Elasticsearch Bulk API, allowing Elastic Beats and other Elasticsearch-compatible shippers to send data to DataStream without reconfiguration.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: elasticsearch
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    port: <numeric>
    address: <string>
    api_version: <string>
    max_body_size: <numeric>
    authentication:
      type: <string>
      username: <string>
      password: <string>
    tokens:
      - token: <string>
        tenant_id: <string>
        expire_date: <numeric>
    reuse: <boolean>
    workers: <numeric>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>
      client_ca_name: <string>
      client_auth_required: <boolean>
      insecure_skip_verify: <boolean>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `elasticsearch`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Connection

Field	Required	Default	Description
`port`	Y	-	TCP port to listen on
`address`	N	`"0.0.0.0"`	Network address to bind
`api_version`	N	`"8.3.2"`	Elasticsearch version string returned in cluster info responses
`max_body_size`	N	`26214400`	Maximum request body size in bytes after decompression (default 25 MB)

Authentication

Field	Required	Default	Description
`authentication.type`	N	`"none"`	Authentication mode: `none`, `basic`, or `bearer`
`authentication.username`	Y*	-	Username for Basic authentication
`authentication.password`	Y*	-	Password for Basic authentication
`tokens`	Y*	-	Array of accepted bearer tokens. Each entry is a plain string or an object (see Multi-Tenancy)

* = Conditionally required: authentication.username and authentication.password are required when authentication.type is basic; tokens is required when authentication.type is bearer.

Multi-Tenancy

Each entry in the tokens list may be a plain string (matched as-is, with no tenant) or an object that binds the token to a tenant and an optional expiry. Both forms can be mixed in the same list.

Field	Required	Default	Description
`tokens[].token`	Y		The accepted token secret. Quote long numeric-looking values to preserve them exactly.
`tokens[].tenant_id`	N	-	Tenant identifier attached to records authenticated with this token. Omit for no tenant.
`tokens[].expire_date`	N	never	Expiry as epoch seconds (e.g. `1924905600`). The token is rejected at and after this instant. Omit for a token that never expires.

A request presenting an expired or unknown token is rejected with 401 and no record is ingested. When a request authenticates with a token that carries a tenant_id, that tenant is written to _vmetric.event.tenant_id; the client IP always stays at _vmetric.event.request. Use _vmetric.event.tenant_id to route or isolate each customer's data downstream (see Routes).

tokens:
  - token: "acme-9f3c...key"
    tenant_id: "acme"
    expire_date: 1924905600
  - token: "globex-7a1b...key"
    tenant_id: "globex"
  - "legacy-plain-token"

note

Token changes take effect on the next collector restart.

Requires authentication.type: bearer. Elastic Beats send tokens without the Bearer scheme prefix; bare tokens are matched the same way.

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Enable TLS encryption
`tls.cert_name`	Y*	`cert.pem`	TLS certificate
`tls.key_name`	Y*	`key.pem`	TLS private key
`tls.passphrase`	N	-	Passphrase for an encrypted private key
`tls.min_tls_version`	N	`tls1.2`	Minimum accepted TLS version (`tls1.0`, `tls1.1`, `tls1.2`, `tls1.3`)
`tls.max_tls_version`	N	-	Maximum accepted TLS version. When unset, the highest mutually supported version is negotiated.
`tls.client_ca_name`	N	-	CA bundle used to verify client certificates (mTLS)
`tls.client_auth_required`	N**	`false`	Require connecting clients to present a valid certificate. When `false`, a client certificate is verified only if one is presented.
`tls.insecure_skip_verify`	N	`false`	Skip peer certificate verification. Use only for testing.

* = Required when tls.status is true.

** = Requires tls.client_ca_name. If tls.client_auth_required is true or tls.client_ca_name is set and the named CA cannot be loaded, the configuration is rejected and the device fails to start.

note

tls.min_version is a deprecated alias for tls.min_tls_version, honored only when tls.min_tls_version is unset. Use tls.min_tls_version in new configurations.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Performance

Field	Required	Default	Description
`reuse`	N	`true`	Enable multi-worker mode
`workers`	N	CPU count	Number of listener workers when `reuse` is enabled (capped at the platform's maximum supported socket count)

Details

The device listens on TCP and exposes an HTTP endpoint that responds to the same API paths used by Elasticsearch. On startup, Beats clients probe several endpoints (GET /, GET /_cluster/health, GET /_nodes, index template paths, ILM policy paths) to verify compatibility. The device acknowledges all probes with well-formed responses so Beats proceed to data ingestion without modification.

The primary ingest path is the Bulk API, available at POST /_bulk and POST /<index>/_bulk. The device parses the NDJSON bulk format (alternating action/source line pairs) and forwards each document to the DataStream pipeline. Action metadata fields (_index, _id, _pipeline, _routing) are injected into the document before forwarding. The delete action is acknowledged in the response but no event is ingested. The update action requires a doc field; scripts, upsert, and scripted_upsert are not supported and are rejected with a per-item error.

The device accepts gzip-compressed request bodies using Content-Encoding: gzip. The body size limit (max_body_size) is enforced after decompression to prevent gzip-bomb attacks. Requests exceeding the limit receive HTTP 413.

The api_version field controls the version number returned in the cluster info response at GET /. Beats use this value to determine API compatibility. The default value 8.3.2 is sufficient for current Beats versions. Change this value only if a specific shipper version requires a different response.

Elasticsearch ingest pipelines can be registered via PUT /_ingest/pipeline/<id> and retrieved via GET /_ingest/pipeline/<id>. Registered pipelines are stored in memory and are not executed during ingestion. The pipeline name is preserved as _pipeline in the forwarded event for downstream processing. The pipeline cache does not survive a collector restart.

Authentication modes:

none: All requests pass through without credential checks. This is the default.
basic: Enforces HTTP Basic authentication. The Authorization: Basic <base64(user:pass)> header is validated against authentication.username and authentication.password. A WWW-Authenticate: Basic realm="Elasticsearch" header is included in all 401 responses.
bearer: Validates the Authorization: Bearer <token> header against the tokens list. Elastic Beats also send tokens as bare values without the Bearer scheme prefix; both formats are accepted. Configuring bearer with an empty tokens list denies all requests. Use none for explicit open access.

Credential changes require a full collector restart to take effect. The authentication middleware builds its validation structures once at startup.

Worker count is capped at the platform's maximum supported socket count. When reuse is disabled, a single worker handles all connections.

Examples

Basic

Creating a basic Elastic device on port 9200...

- id: 1
  name: elastic_ingest
  type: elasticsearch
  properties:
    port: 9200

Basic Authentication

Requiring Basic auth credentials from all senders...

- id: 2
  name: elastic_basic_auth
  type: elasticsearch
  properties:
    port: 9200
    authentication:
      type: basic
      username: "beats"
      password: "s3cret"

Bearer Token Authentication

Validating requests against a list of bearer tokens...

- id: 3
  name: elastic_bearer_auth
  type: elasticsearch
  properties:
    port: 9200
    authentication:
      type: bearer
    tokens:
      - "tok-filebeat-prod-01"
      - "tok-metricbeat-prod-01"

TLS

Enabling TLS encryption on the listener...

- id: 4
  name: elastic_tls
  type: elasticsearch
  properties:
    port: 9243
    tls:
      status: true
      cert_name: "elastic.crt"
      key_name: "elastic.key"
    authentication:
      type: bearer
    tokens:
      - "tok-secure-beat-01"

High-Volume

Scaling listener workers for high-volume ingestion...

- id: 5
  name: elastic_highvol
  type: elasticsearch
  properties:
    port: 9200
    reuse: true
    workers: 8
    max_body_size: 52428800

Synopsis​

Schema​

Configuration​

Device​

Connection​

Authentication​

Multi-Tenancy​

TLS​

Performance​

Details​

Examples​

Basic​

Basic Authentication​

Bearer Token Authentication​

TLS​

High-Volume​

Synopsis

Schema

Configuration

Device

Connection

Authentication

Multi-Tenancy

TLS

Performance

Details

Examples

Basic

Basic Authentication

Bearer Token Authentication

TLS

High-Volume