Skip to main content

Elasticsearch Query Datasets

Elasticsearch Query datasets define what an Elasticsearch Query device reads from an Elasticsearch or OpenSearch cluster: each is a query of the cluster. They attach to devices of type elasticsearch_query, and in Datasets & Profiles the category is labelled Elasticsearch Query. To create and assign datasets through the web interface, see Management. For the dataset and profile model, see Overview.

Collector Types​

Dataset (console title)DefinitionKindDefinition idReads
Elasticsearch Query Documentselasticsearch_query_document_collectorlog57Every document a query matches, in the order of its timestamp, none missed. See Documents.
Elasticsearch Query Aggregationselasticsearch_query_aggregation_collectormetric24150A count of the documents a query matches per window of time, or a metric of their fields, in buckets of tags. See Aggregations.

A device runs any number of either, each with its own index pattern, query, and interval.

Both take these properties:

FieldRequiredDefaultDescription
indexY-The index pattern, as the cluster takes one: names, wildcards, a comma between two, and a leading minus for one left out (logs-*,-logs-debug); aliases and data streams too. Text or a list. A character the cluster refuses in an index name (a space, \, /, ", <, >, |, #, ?) is refused.
intervalN60Seconds between readings, from 1 to 604800 (a week). A duration (5m) is accepted. A reading that failed is tried again within a minute; one the cluster refused waits for its interval.
lagN60 (documents), 0 (aggregations)Seconds behind the time of a reading that it reads to, from 0 to 86400 (a day).
pipeline_nameNthe device'sThe preprocessing pipeline the dataset's records enter.

Queries​

A query — query on the documents dataset, filter_query on the aggregations dataset — is either:

  • Query DSL JSON: a clause ({"match": {"rule.groups": "sshd"}}), or a search body that holds one ({"query": {...}}, as Kibana's Dev Tools write it), as text or as a YAML object; or
  • A query string in Lucene syntax (rule.level:>=10 AND agent.name:web-*), as Telegraf's filter_query and Kibana's Lucene search take one.

Left out, or *, a query matches every document. Refused are:

  • a body that holds what the reading decides itself (size, sort, search_after, pit, aggs, _source, and the like);
  • two clauses side by side (combine them in a bool query);
  • JSON that does not parse, and a query longer than 64 KiB.

The query filters alongside the reading's own window of time.

Documents​

Elasticsearch Query Documents reads every document its query matches, in the order of its timestamp field, and writes each as a record of its own. The delivery is at least once, none missed, for the documents that carry the timestamp field and are indexed within the lag of their timestamp: a document without the field, or one indexed later than that, is not read.

FieldRequiredDefaultDescription
queryNevery documentThe query, as under Queries
timestamp_fieldN@timestampThe date or date_nanos field the documents are read in the order of. A document without it is never read. A Wazuh alert's own time is timestamp; its @timestamp is when Filebeat shipped it.
batch_sizeN1000Documents a page, from 1 to 10000
initial_lookbackN86400Where a dataset with no checkpoint starts, in seconds before its first reading. -1 reads every document the index pattern holds, from the oldest.

How a reading reads​

A reading reads the documents its query matches whose timestamp is past the dataset's checkpoint, and no later than the time of the reading less its lag:

  1. It asks whether any document matches. If none does, the reading opens nothing and the checkpoint moves to its bound, so a reading of a quiet index costs one request.
  2. It opens a point in time of the index pattern, kept alive five minutes between two pages. A search of it sees the indices as they were when it was opened, whatever is indexed or merged afterwards.
  3. It pages through it with search_after, batch_size documents a page, sorted by the timestamp and a tiebreaker: Elasticsearch's _shard_doc, or on OpenSearch the document's _index and _id. A page boundary therefore passes over no document and repeats none, however many documents share a timestamp.
  4. It closes the point in time, a reading cut short included. One that expired while a page was being written is opened again, at most three times a reading, and the reading goes on from its checkpoint.

Checkpoint​

The checkpoint is the timestamp of the last document delivered, with the documents delivered at exactly that timestamp. A reading resumes at the timestamp and passes over the documents it remembers there, so an interrupted reading, a restart in the middle of a backlog, or a failover to another Director node resumes after the last document whose delivery was stored.

  • Stored after delivery. The checkpoint reaches the Director cluster's replicated store once its records are delivered. A crash before then leaves the stored checkpoint where it was, and the next start writes again what was written after it.
  • Past the stream, a record is the route's. A record a pipeline drops, or one bound for a target that is disabled or no longer exists, settles as a delivery would, and the checkpoint moves past it. A document reaches its target at least once only when the route keeps it and its target accepts it.
  • A record that cannot be written ends the reading at once; the next reading reads its page again.
  • 10,000 documents of one timestamp are remembered at most. A restart among more than that writes some of them again, and the reading says so.
  • Per dataset, and it never expires. A device paused for a month resumes where it stopped. A checkpoint the store cannot answer fails the reading, which is tried again: a dataset never starts over while its checkpoint may say where it stopped. A checkpoint that does not decode is logged, and the dataset starts at its initial lookback.
  • Kept whatever the query. Changing a dataset's query or index pattern reads the new query's matches from where the old one stopped.

So a document can be written twice — one delivered just before a crash, before its checkpoint was stored, and some among more than 10,000 of one timestamp — but none is missed on the device's side.

Lag​

The lag is how late a document may be indexed and still be read. A document becomes searchable some time after the time it carries: the shipper's delay, the pipeline's, the index's refresh interval. A reading reads up to its time less the lag, so a document indexed up to lag seconds after its timestamp is read by a later reading.

warning

A document indexed later than the lag after its timestamp, with a timestamp a reading has already passed, is not read. Set the lag to the longest delay the cluster's documents have — a Wazuh manager that buffers alerts while the indexer is down, a Logstash queue that backs up. A larger lag costs only latency.

Records​

The record is the document's _source, every field as it is (its own @timestamp or timestamp kept, numbers as they were written), dated by its timestamp to the nanosecond. It is written to the custom log table, with:

FieldDescription
elasticsearch_query.indexThe document's _index (for a data stream, its backing index)
elasticsearch_query.idIts _id
elasticsearch_query.sortIts sort values: the timestamp in nanoseconds, then the tiebreaker's
event.module, event.datasetelasticsearch_query, elasticsearch_query.document
elasticsearch_query.event.module, elasticsearch_query.event.datasetThe document's own event.module and event.dataset, when it has them, as an Elastic Agent or Beats document does. Every other field of its event is left as it is.

A document whose _source is not an object is left out and logged, and the reading goes on.

Aggregations​

Elasticsearch Query Aggregations counts the documents its filter query matches in a window of time, or computes the metric function over their metric fields, in buckets of the tags' values. Its names are Telegraf's, so a Telegraf elasticsearch_query or opensearch_query configuration carries over.

FieldRequiredDefaultDescription
measurement_nameNelasticsearch_queryThe records' elasticsearch_query.measurement
date_fieldN@timestampThe date field the window is counted by. Its format is not needed.
query_periodNthe intervalThe window's length in seconds, from 1 to 2678400 (a month). 5m and 1h are accepted.
filter_queryNevery documentThe query, as under Queries
metric_fieldsN-The fields metric_function is computed over, at most 32. With none, the records count the documents alone.
metric_functionNavgavg, sum, min, or max
tagsN-Keyword fields whose values cut the matches into buckets, a level each in the order given, at most 8
include_missing_tagNfalsePuts a document without a tag's field in a bucket of missing_tag_value. Otherwise it is in no bucket.
missing_tag_valueNnullThat bucket's value

Windows​

A reading counts the documents whose date_field is in the window [end - query_period, end), where end is the time of the reading less the lag. With tags, each tag is a level of up to a thousand buckets inside the one before; the cluster's own search.max_buckets bounds the whole.

  • With query_period left at the interval, windows follow each other with no gap and no overlap, and no document is counted twice. A reading that runs late or early begins its window where the last counted window ended, so the window is as much longer or shorter; each record carries the window it counts.
  • A window some of whose records were written is counted, and the next begins after it. One none of whose records could be written is counted again by the next reading.
  • More than ten windows late, after the cluster could not be read for longer, a reading counts its own period and logs the span before it that no reading counted.
  • With any other query_period, windows overlap or leave gaps, as the dataset asks, each the period before its own end.

Records​

Each innermost bucket is a record; with no tags, the window is one, a count of zero included. Records go to the metric table, dated at the end of their window:

FieldDescription
event.module, event.dataset, event.kindelasticsearch_query, elasticsearch_query.aggregation, metric
event.start, event.endThe window
observer.*The device and the Director it ran on
elasticsearch_query.measurementmeasurement_name
elasticsearch_query.indexThe index pattern
elasticsearch_query.doc_countThe bucket's documents
elasticsearch_query.tags.<tag>Each tag's value, as text
elasticsearch_query.metrics.<field>_<function>Each metric field's value, named as Telegraf names its fields (rule.level_max)

A metric with no value, such as the average of documents that lack the field, is left out of its record. metric_fields are sent as given: a field the function does not take is a refusal by the cluster, reported as the dataset's.

Console​

In Fleet Management > Datasets, create an Elasticsearch Query Documents or Elasticsearch Query Aggregations dataset. Its configuration step has:

  • Both: Index pattern, Interval (Optional), and Lag (Optional).
  • Documents: Query (Optional), Timestamp field (Optional), Initial lookback (Optional), and Batch size (Optional).
  • Aggregations: Filter query (Optional), Date field (Optional), Query period (Optional), Tags (Optional), Metric fields (Optional), Metric function, Include missing tag, Missing tag value (Optional), and Measurement name (Optional).

Examples​

Reading the Windows security events of the Beats indices from the last week, and counting failed logons per host every minute...

- id: 655200103
  name: elk-winlogbeat
  type: elasticsearch_query
  status: true
  properties:
    urls: https://es.example.com:9200
    username: siem-reader
    password: "$secret{store=corp-vault,ref=elastic/siem-reader}"
  definitions:
    - name: elasticsearch_query_document_collector
      properties:
        index: winlogbeat-*
        query: 'event.provider:"Microsoft-Windows-Security-Auditing"'
        initial_lookback: 604800
    - name: elasticsearch_query_aggregation_collector
      properties:
        measurement_name: failed_logons
        index: winlogbeat-*
        filter_query: 'event.code:4625'
        tags:
          - host.name