Elasticsearch Query

Pull

Synopsis

Director reads one Elasticsearch or OpenSearch cluster through its REST API: an ELK stack, an Elastic Security deployment, or a Wazuh indexer, which is OpenSearch and keeps every Wazuh alert in wazuh-alerts-*. Nothing is installed in the cluster.

Security teams keep their alerts and logs in such a cluster. Moving to another SIEM, or feeding one from the other, means reading those documents out in order, without missing any, and this device does that. It also counts the documents, or computes a metric over them, per interval, as Telegraf's elasticsearch_query and opensearch_query inputs do.

What the device reads is decided by the datasets assigned to it, a query of the cluster each — see Elasticsearch Query Datasets:

• Documents (elasticsearch_query_document_collector): every document a query matches, read in the order of its timestamp, with a checkpoint, so a restart resumes where the last delivery ended.
• Aggregations (elasticsearch_query_aggregation_collector): a count of the documents a query matches in a window of time, or the average, sum, minimum, or maximum of their fields, in buckets of the tags' values.

A device runs any number of either. One device, one cluster: a cluster's nodes are its URLs.

The device dials the cluster and binds no port, so Elasticsearch Query devices never collide on a port. One Director node reads each device: the device takes part in the cluster's ownership lease, since two would read every document twice and keep a checkpoint each.

The Elastic device works the other way round: it takes the documents a shipper pushes to it through the Bulk API. To write documents into a cluster, use the Elasticsearch target.

note

The Elasticsearch Query device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows Elasticsearch Query devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.

The device does not run in simulation. Its records are a cluster's own documents, of the index's own shape, and counts of them, so a generated sample cannot stand in for them. See Simulation.

Account

The account needs to read the datasets' indices: the read index privilege on Elasticsearch, the read action group on OpenSearch. These open, search, and close a point in time of the indices. The device reads; it changes nothing.

Asking the cluster what it is (GET /) needs a monitor privilege the account may lack. A refusal there is still an answer from the cluster, so the device stays Connected.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: elasticsearch_query
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    urls: <string | string[]>
    engine: <string>
    username: <string>
    password: <string>
    api_key: <string>
    bearer_token: <string>
    timeout: <numeric>
    tls:
      insecure_skip_verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be elasticsearch_query
tags	N	-	Optional tags
pipelines	N	-	Optional pre-processor pipelines
status	N	true	Enable/disable the device

Cluster

Field	Required	Default	Description
urls	Y	-	A URL, or a list of up to 16: the nodes of one cluster, or a load balancer in front of them. url names one when urls is not given. Each is http:// or https://, all of one scheme, with a host and an optional path prefix (https://proxy.example.com/elasticsearch). A wildcard host, a username or password, a query, and a fragment are refused.
engine	N	elasticsearch	elasticsearch (7.12 or later) or opensearch (2.4 or later; a Wazuh indexer is OpenSearch). The two name their point in time API differently, so with the wrong engine the documents datasets cannot be read.
timeout	N	30	Seconds a request may take whole — the connection, the handshake, the request, and the answer read to its end — from 1 to 300. A duration of whole seconds (45s, 2m) is accepted.

A request goes to the URL that answered last, and to the next when that one does not answer. A refusal is the cluster's answer, and is not asked of another node.

Authentication

Field	Required	Default	Description
username	N	-	Basic authentication, with password
password	N	-	The user's password. A password with no username is refused.
api_key	N	-	An Elasticsearch API key, sent as Authorization: ApiKey: the encoded value Elasticsearch returns, or its id and api_key as id:key
bearer_token	N	-	Sent as Authorization: Bearer <token>: an Elasticsearch service account token, or an OpenSearch JWT

• One authentication at most. A username and password, an API key, or a bearer token; a device that sets two of them is refused. A device that sets none signs in with nothing.
• Each takes plain text, ${ENV_VAR}, or $secret{...} ($secret{id=...} or $secret{store=...,ref=...}), resolved at every request, so a credential rotated in the vault is sent from the next request on. The secret store's own cache, five minutes by default, keeps a request from asking the store every time.
• OpenSearch takes no API key. Its security plugin takes basic authentication or a bearer token. A device that gives OpenSearch an API key says so in its log when it starts.
• An error names the property, never its value. A value with a line break in it is not sent.

warning

Over http:// the credentials cross the network in cleartext with every request, and anyone who reads them can sign in to the cluster with them. The device says so in its log when it starts, and runs. Use an http:// URL with credentials only where the network between the Director and the cluster is trusted.

TLS

The scheme of the URLs decides whether the requests use TLS: https:// does, http:// does not. These settings secure https:// URLs and are not used otherwise. tls.status, when given, has to agree with the scheme.

Field	Required	Default	Description
tls.insecure_skip_verify	N	false	true accepts any certificate. Set the CA instead where you can.
tls.ca_name	N	the system trust store	The CA that signed the cluster's certificate, such as the one Elasticsearch generated or a Wazuh indexer's root CA
tls.server_name	N	the URL's host	The name the certificate is verified for
tls.cert_name	N	-	A client certificate, for a cluster that identifies the client by its certificate
tls.key_name	N	-	The client certificate's private key. A certificate and its key are given together, or neither.
tls.passphrase	N	-	The private key's passphrase
tls.min_tls_version	N	tls1.2	tls1.0, tls1.1, tls1.2, or tls1.3
tls.max_tls_version	N	-	No cap

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Datasets

The device's definitions are its datasets: elasticsearch_query_document_collector and elasticsearch_query_aggregation_collector. In the console they are the datasets and profiles assigned to the device. For their properties, how documents are read in order, and their records, see Elasticsearch Query Datasets.

A device with no dataset checks every minute that its cluster answers.

Requests

• JSON over HTTP or HTTPS, asked for compressed. Connections are kept between requests, so a reading of many pages does not make a TLS handshake for each.
• A proxy named in the Director's environment (HTTPS_PROXY, HTTP_PROXY, NO_PROXY) is used to reach the cluster.
• A redirect is not followed: it is the cluster's answer. Following it would send the credentials on.
• An answer is read up to 256 MiB. A page larger than that fails, saying to lower the dataset's batch_size.
• Partial results are refused. An answer in which a shard failed or timed out fails the reading, rather than passing over its documents as if they had been read.
• An index pattern that matches no index is nothing to read. An index named without a wildcard that does not exist is refused.

Connection States

What happened	State
A reading, or the device's question of what the cluster is, answered	Connected
Credentials that cannot be resolved, or a configuration that does not validate	Configuration error
Credentials the cluster refuses (401)	Authentication error
A certificate that does not verify	Certificate error
No URL answers, or none in time	Connection error

• A configuration error — no URL, a bound passed, a value that is not a number or a switch — stops the device at start, and the message names the property. A dataset whose settings cannot be read is left out with a warning, and the rest of the device runs; a device whose every dataset is left out stops with Configuration error.
• A refusal is the dataset's. An index that does not exist (404), a query the cluster does not take (400), or an index the account may not read (403) is the cluster answering, so the device stays Connected. Its state's message names each dataset refused, and why, until that dataset reads again. The dataset is asked again every interval, so an index a rollover creates at midnight is read from then on.
• The wrong engine. A point in time asked for at a path the cluster has no API at says which engine the device names, since the cluster most often runs the other. The device also says once, when it starts, if the cluster calls itself the other engine.
• Any other failure of one reading — a shard that failed, a page that could not be written — is the dataset's alone, logged, and moves nothing.

Configuration Changes

Any change to the device, or to a dataset assigned to it, restarts the device; its documents datasets resume from their checkpoints. An edit that no longer validates restarts it into Configuration error.

• Credentials are compared as they are configured, which for a secret is its name, not its value. A secret rotated in its store is sent from the next request on and restarts nothing.
• TLS material is compared by a digest of what it reads. A CA rotated where it is kept restarts the device. One that cannot be read at the moment, with the settings unchanged, is logged and restarts nothing.

Console

In the console, go to Fleet Management > Devices and create an Elasticsearch Query device. The wizard's steps are:

1. General settings, as on every device.
2. Connection settings:
   • URL, with HTTPS or HTTP selected beside the address, and Other nodes (Optional): more nodes of the same cluster, reached over the URL's protocol, at most 15.
   • Search engine (Elasticsearch or OpenSearch) and Timeout.
   • For HTTPS, the TLS settings: whether to skip certificate verification, the CA certificate, Server name (Optional), the minimum and maximum TLS versions, and a Client certificate: Certificate (Optional), Private key (Optional), and Passphrase (Optional).
   • Authentication: an Authentication method of None, Basic (Username and Password (Optional)), API key, or Bearer (Bearer token).
3. The datasets or profile the device reads. A device receives datasets or a profile, never both.
4. Review and create.

The wizard warns when credentials would go over HTTP (The credentials are sent in cleartext) and when OpenSearch is given an API key (OpenSearch takes no API key).

Examples

ELK Stack Alerts with an API Key

Moving an ELK cluster's alerts to another SIEM: every document of the pattern but the test index, from the oldest, signing in with an API key kept in a vault...

devices: - id: 655200101 name: elk-alerts type: elasticsearch_query status: true properties: urls: - https://es-1.example.com:9200 - https://es-2.example.com:9200 engine: elasticsearch api_key: "$secret{store=corp-vault,ref=elastic/siem-reader}" tls: ca_name: elk-ca.pem definitions: - name: elasticsearch_query_document_collector properties: index: alerts-*,-alerts-test query: | {"bool": {"filter": [{"terms": {"event.kind": ["alert", "signal"]}}]}} lag: 120 initial_lookback: -1

Wazuh Indexer

Reading Wazuh alerts of level 7 and over by their own timestamp field, and counting authentication failures per agent every five minutes, with the highest rule level of each...

devices: - id: 655200102 name: wazuh-indexer type: elasticsearch_query status: true properties: urls: https://wazuh-indexer.example.com:9200 engine: opensearch username: siem-reader password: ${WAZUH_INDEXER_PASSWORD} tls: ca_name: wazuh-root-ca.pem server_name: wazuh-indexer definitions: - name: elasticsearch_query_document_collector properties: index: wazuh-alerts-* timestamp_field: timestamp query: "rule.level:>=7" interval: 30 - name: elasticsearch_query_aggregation_collector properties: measurement_name: wazuh_alerts index: wazuh-alerts-* date_field: timestamp interval: 300 filter_query: "rule.groups:authentication_failures" metric_fields: - rule.level metric_function: max tags: - agent.name include_missing_tag: true missing_tag_value: unknown