Amazon Data Firehose

Amazon AWS HTTP Endpoint

Synopsis

The Amazon Data Firehose device (type: awsfirehose) is an HTTP endpoint that an Amazon Data Firehose stream delivers to through its HTTP endpoint destination. Director listens, the stream posts batches of records, and every record is written. The device implements version 1.0 of that destination's delivery request and response specification.

AWS services that do not write to a bucket deliver this way: CloudWatch Logs through a subscription filter, AWS WAF logs, VPC Flow Logs, EventBridge events, or what an application puts with PutRecord. Director takes them as they are delivered, with no bucket, no SQS queue and no AWS credentials. AWS pushes, and nothing is polled: Director calls no AWS API. What the stream needs is a URL it can reach — see Reaching Director from AWS.

A record is written as it came, and the device's pipeline parses it. The records of a CloudWatch Logs subscription are the exception: the device takes them apart into their log events — see CloudWatch Logs.

Firehose's Splunk destination needs no device of this type. It delivers to a Splunk HEC device.

note

The Amazon Data Firehose device needs Director 2.3.1 or later.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: awsfirehose
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    path: <string>
    access_key: <string>
    record_format: <string>
    record_metadata: <boolean>
    max_body_size: <numeric>
    timezone: <string>
    reuse: <boolean>
    workers: <numeric>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `awsfirehose`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Endpoint

Field	Required	Default	Description
`address`	N	`0.0.0.0`	The address to listen on
`port`	N	`443`	From `1` to `65535`. Firehose delivers to port `443` alone. Use another port only behind a load balancer or a port mapping, and listen on the port that forwards to the device
`path`	N	`/`	The URL path the stream posts to. It begins with a slash, and is a path alone: a query, a fragment or white space is refused. The query a stream's URL is configured with is not part of it and is not matched
`access_key`	N	-	What a request's `X-Amz-Firehose-Access-Key` header has to be, up to 4096 bytes, the longest a stream sends. A secret: name an environment variable or a vault secret. Left out, the endpoint takes every request that reaches it. A key that is named and resolves to nothing is refused, so a secret that went missing does not open the endpoint
`record_format`	N	`auto`	What is made of a record: `auto`, `lines` or `raw`. See Records
`record_metadata`	N	`false`	Write every event as a JSON object of the event (`data`) and what its request says of it: the request's id, the stream's ARN, and its common attributes. See Record Metadata
`max_body_size`	N	`67108864`	The largest request taken, in bytes, as it is sent and as it decompresses: from `1048576` (1 MiB) to `134217728` (128 MiB). The default, 64 MiB, is the largest request a stream sends

A device has one path. To take two paths, use two devices on two ports, or one device and a pipeline that tells the streams apart by their ARN — see Records.

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Serve HTTPS. Requires `tls.cert_name` and `tls.key_name`
`tls.cert_name`	Y*	-	The certificate the endpoint presents. Firehose accepts only one that a public certificate authority signed, whose name is the URL's host
`tls.key_name`	Y*	-	The certificate's private key
`tls.passphrase`	N	-	The passphrase of the private key, when it is encrypted
`tls.min_tls_version`	N	`tls1.2`	The lowest TLS version the endpoint accepts
`tls.max_tls_version`	N	-	The highest TLS version the endpoint accepts. No cap when left out

* = Required when tls.status is true.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Advanced Configuration

Field	Required	Default	Description
`timezone`	N	-	The time zone written with the device's records
`reuse`	N	`true`	Share the port between listeners (`SO_REUSEPORT`). With `false` the port is one listener's, and `workers` is `1`
`workers`	N	Scales with CPU count	How many listeners share the port, from `1` to `4`. Left out, it is `1` up to 16 cores, `2` up to 32, `3` up to 64 and `4` beyond, and reduced further on memory-constrained hosts

The device's request timeouts are the HTTP device's and are not configured.

A configuration that cannot run stops the device with Configuration error and says why: a path that is not valid, a body size out of its bounds, a record format that is none of the three, an access key longer than 4096 bytes or one that resolves to nothing, an address that is not valid, or TLS without its certificate or key.

A change to the address, the port, TLS, the path, the access key, the body size, the record format, the record metadata or the workers restarts the listener. The access key is compared by its value, so a rotated secret takes effect the next time Director compares the device's configuration. A secret that cannot be read at that moment, such as a vault that does not answer, changes nothing: the endpoint goes on with the key it read.

Reaching Director from AWS

This is the part most often got wrong. Firehose asks three things of an HTTP endpoint, and none of them can be configured away:

HTTPS, on port 443. The stream's URL has no port.
A certificate that a public certificate authority signed, whose name is the URL's host. A private CA's certificate and a self-signed one are refused.
An address AWS can reach. Firehose calls the endpoint over the internet. It has no IP range of its own for HTTP endpoints, so an ingress allowlist has to admit AWS's published address ranges rather than one region's.

So there are two ways to deploy the device:

Director serves TLS itself: tls.status: true with a public certificate, port: 443, and a public address or a port forward to it.
A load balancer in front: an Application Load Balancer, an API gateway or a reverse proxy holds the certificate on port 443 and forwards plain HTTP to the device (tls.status: false, any port). This is the way for a cluster: every node runs the device, and the load balancer shares the requests between them.

A load balancer's health check is a GET to the device's path, answered 200 with {}; it asks for no access key. A GET to another path is answered 404, and a method that is neither POST nor GET 405.

On the stream's side, in the AWS console, choose the destination HTTP endpoint and set:

the URL, https://<host><path>;
the access key, the device's access_key;
content encoding, GZIP or disabled — the device reads both;
the retry duration.

The stream's parameters are sent with every request as its common attributes — see CloudWatch Logs.

Records

A request carries up to 10,000 records. A gzip-compressed record is decompressed first, whatever the format. record_format decides what is made of each:

`record_format`	A record becomes
`auto`	One event, without the line end its producer closed it with. A record that is the payload of a CloudWatch Logs subscription becomes its log events
`lines`	One event per line. An empty line is no event. For a producer that puts several lines into one record
`raw`	One event, with its line ends: nothing is trimmed, and a CloudWatch Logs payload is one event

An empty record is no event, nor is one that decompresses to nothing. A record is written as it came, and the device's pipeline parses it.

Every event's request (_vmetric.event.request) is the ARN of the stream that delivered it, from the X-Amz-Firehose-Source-Arn header. That is what tells apart the events of two streams that deliver to one device. A request that names no stream, or one longer than 512 bytes, is written with the address of its peer.

Every event is dated when its request is received, apart from a CloudWatch log event.

CloudWatch Logs

A subscription filter delivers a gzip-compressed JSON document per batch: the log group, the log stream and the log events. With record_format: auto each log event is written as one event:

{
  "id": "38378212459134562177402895230137412345678901234567890123",
  "timestamp": 1790000000123,
  "message": "START RequestId: 6f1e…",
  "logGroup": "/aws/lambda/orders",
  "logStream": "2026/10/03/[$LATEST]abc",
  "owner": "123456789012",
  "subscriptionFilters": ["to-firehose"],
  "commonAttributes": { "environment": "production" }
}

The field names are CloudWatch's own.
The event is dated by CloudWatch (timestamp, to the millisecond), not by when it arrived.
message is the log line as CloudWatch holds it. A line that is JSON is JSON inside a string: the pipeline's json processor on message unfolds it.
commonAttributes are the stream's parameters, written whole, when the stream is configured with any. Without record_metadata they are written on CloudWatch log events alone: any other record is written as it came, and has no place for them. With it, every event has them, as common_attributes — see Record Metadata.
A control message, which CloudWatch sends to check that the destination can be reached, is acknowledged and written as nothing.
It makes no difference whether the stream decompresses CloudWatch's records or delivers them compressed, and several batches in one record are each taken apart.

A record that only looks like a subscription's — it begins as one and goes on as something else, or it has a subscription's message type and no list of log events — is written as it came.

Bounds on Decompression

What is decompressed is bounded twice. One record is decompressed to at most 64 MiB, and the records of one request to at most 16 times max_body_size in all (1 GiB at the default). A compressed record past either is written as it was delivered, one event in every format, and the device logs a warning with how many there were.

Logs compress about ten to one, so a request of CloudWatch batches stays well inside both. The bounds stop a request of small records that each unfold to the most a record may from having the device decompress and write tens of gigabytes. A request is written a record at a time: the device holds what one record unfolds to, not what all of them do.

An event larger than the device's buffer can ever admit (max_buffer_size, or a shard's share of it) is not written. The device logs a warning with its size and takes the request with the rest of its events, since refusing it would have the stream send the same event for as long as it retries.

Record Metadata

With record_metadata: true every event is written as a JSON object: what the device writes without the option is its data, and what the request says of it is beside it. It is off by default.

{
  "data": "user=ada action=login",
  "request_id": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "source_arn": "arn:aws:firehose:eu-west-1:123456789012:deliverystream/orders",
  "common_attributes": { "environment": "production", "team": "shop" }
}

Field	What it is
`data`	The event as the device writes it without the option: the record, decompressed, one line of it, or a CloudWatch log event's object. Text when it is UTF-8, and base64 when it is not. A record that is JSON is JSON inside a string: the pipeline's `json` processor on `data` unfolds it
`data_encoding`	`base64` when `data` is base64. Left out for text
`request_id`	The request's id. A retry of the request has the same one
`source_arn`	The ARN of the stream that delivered it (`X-Amz-Firehose-Source-Arn`). Left out when the request names none, or one longer than 512 bytes
`common_attributes`	The stream's parameters, by their names. Left out when the stream is configured with none

Every event of a request has its request's fields: every line with lines, and every log event of a CloudWatch Logs payload.
The object is one line.
An event is dated, and has its request, as it is without the option: a CloudWatch log event by CloudWatch, any other when its request is received.

Answers and Retries

The device answers every delivery request as the specification asks: a JSON body with the request's id and the time.

Status	When	What the stream does
`200`	Every event of the request is written, apart from an event too large for the device's buffer, which is skipped with a warning — see Bounds on Decompression	Takes the request as delivered, and does not send a skipped event again
`400`	The body is no delivery request (not JSON, or a record that is not base64), it names no request id or one longer than 256 bytes, the id in its body is not its header's, or its compressed body is corrupt	Sends it again for its retry duration, then backs it up to S3 when a backup bucket is configured
`401`	The access key is not the device's. Checked before the body is read	The same
`413`	The request is larger than `max_body_size`, as it is sent or as it decompresses	Does not send it again
`415`	A content encoding the device does not read	Sends it again
`503`	The device's buffer had no room within two minutes	Sends it again
`404`, `405`	Another path, or a method that is neither `POST` nor `GET`	Sends it again

Nothing of a refused request is written. A refusal says why in errorMessage, which the stream writes to its error log in CloudWatch. What failed inside Director is in Director's log, not in the answer.

A request that is sent again is not written twice. A stream sends a request again under the id of its first try, both when it was refused and when it was taken and its answer was lost or late. The device remembers the requests it was sent and how many events of each it wrote: a retry of a request that was written is answered 200 and writes nothing, and a retry of one that was cut short by a 503 writes the rest. Two tries of one request that arrive together are written once.

The limits of that, which is why delivery is at least once:

A request is remembered for 3 hours (a stream retries for 2 at most), and the device remembers 50,000 of them, the oldest forgotten first.
It is remembered in memory, by one Director. A listener that restarts starts with none, and in a cluster a retry that the load balancer hands to another node is written by that node.

Connection States

Connected: the configuration is read and the port can be bound.
Connection error: the port cannot be bound, because another device holds it or Director may not bind a port below 1024.
Configuration error: a configuration that cannot run — see Advanced Configuration.

On stop, the listener takes no new request and waits up to 5 seconds for the ones it is answering. One that is not done by then is closed, and the stream sends it again.

Console

In the console, go to Fleet Management > Devices and choose Amazon Data Firehose, under Amazon Web Services. The wizard's steps after General settings are:

Protocol settings: Listen address, Port, Path, Access key, and Record format — Automatic, One event per line, or One event per record — and Record metadata (Disabled or Enabled, off by default). Then TLS encryption, with Certificate, Private key, Passphrase, and the minimum and maximum TLS version. The step's description says what certificate Firehose accepts.
Advanced settings: Socket address reuse, Workers, Maximum request size, and the Simulation section.

With no Director or cluster at version 2.3.1 or later selected, the wizard shows Amazon Data Firehose devices are unavailable on directors or clusters below version 2.3.1, and the device cannot be created.

Simulation

The device can be simulated as the other listeners are — see Device Simulation. In standalone mode the device binds nothing. In inject mode its collector starts and binds its port as usual. Either way its generated records carry a stream's ARN as their request: arn:aws:firehose:us-east-1:000000000000:deliverystream/simulation-<sample>.

Examples

Serving TLS on Director

Taking CloudWatch Logs from a Firehose stream, with Director serving a public certificate on port 443 and the access key read from an environment variable...

devices:
  - id: 512400400
    name: firehose-cloudwatch
    type: awsfirehose
    status: true
    properties:
      address: "0.0.0.0"
      port: 443
      path: "/firehose/cloudwatch"
      access_key: ${FIREHOSE_ACCESS_KEY}
      tls:
        status: true
        cert_name: "firehose.example.com.crt"
        key_name: "firehose.example.com.key"
      record_format: auto

Behind a Load Balancer

Taking AWS WAF logs, one JSON object per line, behind a load balancer that holds the certificate on port 443 and forwards plain HTTP to port 8443 on every node...

devices:
  - id: 512400401
    name: firehose-waf
    type: awsfirehose
    status: true
    properties:
      address: "0.0.0.0"
      port: 8443
      path: "/firehose/waf"
      access_key: ${FIREHOSE_ACCESS_KEY}
      record_format: lines
      max_body_size: 16777216
      workers: 2

Synopsis​

Schema​

Configuration​

Device​

Endpoint​

TLS​

Advanced Configuration​

Reaching Director from AWS​

Records​

CloudWatch Logs​

Bounds on Decompression​

Record Metadata​

Answers and Retries​

Connection States​

Console​

Simulation​

Examples​

Serving TLS on Director​

Behind a Load Balancer​