HTTP

Network Webhook

Synopsis

Creates a target that sends log data to HTTP/HTTPS endpoints using configurable methods, formats, and authentication. Supports batching, compression, retry logic, and connection pooling for reliable delivery to web services, APIs, and webhooks.

Schema

- name: <string>
  description: <string>
  type: http
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    method: <string>
    format: <string>
    content_type: <string>
    headers:
      <key>: <value>
    batch_size: <numeric>
    timeout: <numeric>
    connect_timeout: <numeric>
    socket_timeout: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    compression: <boolean>
    keep_alive: <boolean>
    follow_redirects: <boolean>
    pool_max: <numeric>
    pool_max_per_route: <numeric>
    authentication:
      type: <string>
    username: <string>
    password: <string>
    token: <string>
    header:
      key: <string>
      value: <string>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be http or https
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

HTTP Connection

Field	Required	Default	Description
url	Y	-	Destination URL (must use http:// or https:// scheme)
method	N	POST	HTTP method: GET, POST, PUT, PATCH, DELETE, HEAD
format	N	json	Output format: json, json_batch, form, message
content_type	N	auto	Content-Type header (auto-detected from format)
headers	N	-	Custom HTTP headers as key-value pairs
max_bytes	N	10485760 (10 MB)	Maximum size of a single request body or batch, in bytes

note

max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:

• An explicit max_bytes: 0 IS honoured and removes the ceiling on targets that have no ceiling of their own. The value is read with Int64, which returns what you wrote — unlike max_size, which is read with MustInt64 and silently substitutes its default for a zero. Where a target has a fixed ceiling of its own, its page says so and a zero is clamped to that ceiling instead.
• A negative value is rejected at configuration time with max_bytes must not be negative.

The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.

Request Settings

Field	Required	Default	Description
batch_size	N	1000	Maximum number of events per batch
timeout	N	60	Request timeout in seconds
connect_timeout	N	10	Connection establishment timeout in seconds
socket_timeout	N	10	Socket read/write timeout in seconds
compression	N	false	Enable gzip compression
keep_alive	N	true	Enable HTTP keep-alive connections
follow_redirects	N	true	Follow HTTP redirects

Retry Configuration

Field	Required	Default	Description
max_retries	N	0	Maximum retry attempts on transient errors (dial failures, DNS errors). Use -1 for unlimited retries on transient errors; values less than -1 are rejected.
retry_delay	N	1	Delay between retries in seconds

Connection Pool

Field	Required	Default	Description
pool_max	N	50	Maximum idle connections in pool
pool_max_per_route	N	25	Maximum connections per route

Authentication

Field	Required	Default	Description
authentication.type	N	none	Authentication type: none, basic, bearer, header
username	N*	-	Username for basic authentication
password	N*	-	Password for basic authentication
token	N*	-	Token for bearer authentication
header.key	N*	-	Header name for header authentication
header.value	N*	-	Header value for header authentication

* = Required when using the corresponding authentication type.

warning

authentication.type is nested, but the credential fields are not. username, password, token and the header block sit at the top level of properties, next to url. A configuration that nests them under authentication: is rejected with username and password are required for basic authentication, or the matching message for bearer and header, and the target never starts.

header and headers are two different settings. header carries the one authentication header, and headers carries custom headers added to every request.

TLS Configuration

Field	Required	Default	Description
tls.status	N	false	Enable TLS client certificate authentication

TLS material is resolved through the shared client builder, so these keys mean the same thing on every target that uses it. They are nested under a tls: block.

Field	Required	Default	Description
tls.verify	N	true	Verify the server certificate. On by default — set it to false only to accept an untrusted certificate, and only where you control the network path
tls.server_name	N	-	SNI hostname override for the handshake. Use it when the certificate's name does not match the address you connect to
tls.ca_name	N	-	CA bundle used to verify the server certificate. When unset the host trust store is used; when set it replaces the host trust store rather than adding to it
tls.cert_name	N*	-	Client certificate, for mutual TLS
tls.key_name	N*	-	Client private key, for mutual TLS
tls.passphrase	N	-	Passphrase for an encrypted client private key
tls.min_tls_version	N	tls1.2	Lowest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_version	N	tls1.3	Highest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)

* cert_name and key_name are individually optional but must be supplied together — a certificate without its key, or a key without its certificate, is a configuration error.

Normalization

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The https target type is an alias for http and shares an identical configuration schema. Both target types accept URLs with either http:// or https:// scheme regardless of the type string used.

Output Formats

The format field determines how events are sent to the HTTP endpoint:

Format	Content-Type	Description
json	application/json	Each event sent as separate JSON object request
json_batch	application/json	All events sent as JSON array in single request
form	application/x-www-form-urlencoded	Events encoded as form data
message	application/json	Raw message content, newline-separated

The message body is plain text, but it is sent with the JSON content type unless you set one. Set content_type: "text/plain" alongside format: message when the endpoint expects plain text.

Authentication Types

Basic Authentication: Uses HTTP Basic Auth with username and password encoded in the Authorization header.

Bearer Authentication: Sends a token in the Authorization header as Bearer <token>.

Header Authentication: Adds a custom header with configurable key and value, useful for API keys.

Compression

When compression: true is enabled, the request body is gzip-compressed and the Content-Encoding: gzip header is set. This reduces bandwidth usage for high-volume data transmission.

Connection Pooling

The HTTP client maintains a connection pool for efficient connection reuse. Tune pool_max and pool_max_per_route based on expected concurrency and target endpoint capacity.

warning

Setting tls.verify: false disables certificate verification and is not recommended for production environments.

Examples

Basic Webhook

Sending events to a webhook endpoint using default JSON format...

targets: - name: webhook type: http properties: url: "https://webhook.example.com/events"

With API Key Authentication

Using header-based authentication for API key...

targets: - name: api_endpoint type: http properties: url: "https://api.example.com/logs" authentication: type: header header: key: "X-API-Key" value: "${API_KEY}"

With Bearer Token

Using OAuth bearer token authentication...

targets: - name: oauth_api type: http properties: url: "https://api.example.com/ingest" authentication: type: bearer token: "${BEARER_TOKEN}"

With Basic Authentication

Using HTTP Basic authentication with username and password...

targets: - name: basic_auth_endpoint type: http properties: url: "https://api.example.com/logs" authentication: type: basic username: "${HTTP_USERNAME}" password: "${HTTP_PASSWORD}"

Batch JSON

Sending events as JSON array for efficient batch processing...

targets: - name: batch_api type: http properties: url: "https://api.example.com/batch" format: json_batch batch_size: 500 compression: true

High Volume with Retries

Optimized for high-volume delivery with retry logic and connection pooling...

targets: - name: high_volume_http type: http properties: url: "https://collector.example.com/events" format: json_batch batch_size: 1000 compression: true max_retries: 3 retry_delay: 2 timeout: 30 pool_max: 100 pool_max_per_route: 50 authentication: type: bearer token: "${COLLECTOR_TOKEN}"

With Custom Headers

Adding custom headers for routing or metadata...

targets: - name: custom_headers type: http properties: url: "https://api.example.com/logs" headers: X-Source: "datastream" X-Environment: "production" X-Tenant-ID: "tenant-123"

With Client Certificate (mTLS)

Using mutual TLS with client certificate authentication...

targets: - name: mtls_endpoint type: http properties: url: "https://secure-api.example.com/events" tls: status: true verify: true cert_name: "client-cert.pem" key_name: "client-key.pem" min_tls_version: "tls1.2"

PUT Method

Using PUT method for REST API updates...

targets: - name: rest_update type: http properties: url: "https://api.example.com/resources/logs" method: PUT format: json

Form Data

Sending data as URL-encoded form...

targets: - name: form_endpoint type: http properties: url: "https://legacy.example.com/submit" format: form method: POST

With Field Normalization

Applying ECS normalization before sending to HTTP endpoint...

targets: - name: normalized_http type: http properties: url: "https://siem.example.com/events" format: json_batch field_format: ecs compression: true

Troubleshooting

This section covers the errors you are most likely to see with the http target, what causes each one, and how to fix it. Everything here applies to the https target as well, since it is the same target under a second type name.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason:, or after the last colon, is the actual cause. For this target that cause is either a status code the endpoint returned, or the network error the request failed with.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

Nothing is sent while the target starts up, so a wrong host, a closed port or a rejected credential is only discovered on the first flush. Until then the target can look healthy.

See Target Delivery Errors for how Director logs and retries target failures.

What the endpoint needs

Match the error you see against this table first. Each row is a prerequisite on the receiving side, not a permission you grant in a cloud console.

Prerequisite	Error when it is missing
A route at url that accepts the configured method	received status code 404 or received status code 405
A credential the endpoint accepts, matching authentication.type	received status code 401 or received status code 403
A body encoding the endpoint parses, set by format and content_type	received status code 400, 415 or 422
A body below the limit of the endpoint and of any proxy in front of it	received status code 413
DNS resolution and TCP reachability of the host in url, from the Director host	dial tcp: lookup ...: no such host, connect: connection refused, i/o timeout
A server certificate the Director host trusts, for https://	x509: certificate signed by unknown authority
A client certificate the server accepts, for mutual TLS	remote error: tls: bad certificate or tls: certificate required
TLS material readable by the service account and stored under the Director installation directory	could not be resolved (env/vault token, inline PEM, or a path under the service root)

"username and password are required for basic authentication"

[Error] [director] [target-<target id>] [basic_auth_endpoint] ValidateConfig failed for target "basic_auth_endpoint": username and password are required for basic authentication
ValidateConfig failed for target "oauth_api": token is required for bearer authentication
ValidateConfig failed for target "api_endpoint": header key and value are required for header authentication

Cause: the credential fields are nested under authentication: together with type. Only type is read from there. username, password, token and the header block are read one level up, directly under properties. When they are nested, the target sees an authentication type with no credential and refuses the configuration. Which of the three messages you get depends on the type you chose.

Fix: keep type under authentication: and move the value fields up, next to url:

properties:
  url: "https://logs.example.com/ingest"
  authentication:
    type: basic
  username: "${HTTP_USERNAME}"
  password: "${HTTP_PASSWORD}"

For bearer, put token at that same level. For header, put a header block with key and value there. Note that header and headers are two different settings. header carries the one authentication header, and headers carries custom headers added to every request.

The same shape applies to a target created in the web interface. If the saved configuration nests the credentials under the authentication block, open the target's YAML and move them up.

Data impact: none. This is a configuration error, so the target never starts and nothing is sent. Incoming data waits in the Director queue. The configuration is re-checked about every 30 seconds, so the fix is picked up without a restart.

"received status code 401" or "received status code 403"

Sender worker 0 execute() failed for oauth_api: failed to send HTTP request (non-retryable): received status code 401 from server: {"error":"token expired"}

Cause: the endpoint refused the credential. 401 means it did not accept the credential presented. 403 means it accepted the credential but does not allow the write. The text after from server: is the endpoint's own response body, so read that first. Common triggers are a rotated or expired token, a ${...} reference that does not resolve in the Director service environment, and a header name the API does not use for its API keys.

Fix: replace the credential, and confirm the type matches what the API expects. Reproduce the same request by hand from the Director host to prove the credential works before putting it back in the target.

Data impact: none, as long as you fix it inside the queue retention window. The (non-retryable) label means only that the worker did not retry the request itself. The payload returns to the queue and is redelivered roughly every 5 seconds until the endpoint accepts it, so expect the queue to grow.

"record rejected by target: received status code 400"

Sender worker 2 execute() failed for webhook: failed to send HTTP request (non-retryable): record rejected by target: received status code 400 from server: {"error":"malformed json at offset 12"}
Sender worker 2 deterministic failure for webhook after 4 attempts — dropping (giving up): failed to send HTTP request (non-retryable): record rejected by target: received status code 400 from server: {"error":"malformed json at offset 12"}

Cause: the endpoint received the request and refused the body. A 422 behaves the same way. With json or json_batch the body is one JSON array for the whole batch, and each element is an object with a message field holding the event as a string and a timestamp field. An endpoint that expects one object per request, a bare array of events, or its own envelope answers 400.

Fix: choose the format the endpoint documents, set content_type to match it, and reshape the payload in a pipeline when the endpoint needs its own envelope. Testing with batch_size: 1 keeps the rejected body small enough to read in the endpoint's own logs.

Data impact: data is lost. A rejection of this kind is treated as permanent, so the payload is dropped after 4 deliveries. The dropping (giving up) line is the last you see of those events.

"received status code 415", or a message body refused as invalid JSON

Sender worker 1 execute() failed for webhook: failed to send HTTP request (non-retryable): received status code 415 from server:

Cause: the content type does not match the body. With format: message the body is the raw messages joined by newlines, but the request carries the JSON content type unless you set one yourself. An endpoint that expects plain text answers 415, and an endpoint that validates the body as JSON answers 400.

Fix: set content_type explicitly whenever you set format:

properties:
  url: "https://logs.example.com/ingest"
  format: message
  content_type: "text/plain"

Data impact: a 415 is retried until you fix it. Retries continue while Director runs and while the records are inside the queue retention limit, which is queue.limit seconds, 48 hours by default. With the In-memory storage tier they do not survive a Director restart. A 400 is a rejection, so those events are dropped after 4 attempts. Set the content type before a backlog builds up.

"received status code 413", or "record size ... exceeds configured max_bytes"

Sender worker 3 execute() failed for batch_api: failed to send HTTP request (non-retryable): record rejected by target: received status code 413 from server: {"error":"payload too large"}
Sender worker 1 execute() failed for webhook: record rejected by target: record size 12582976 exceeds configured max_bytes 10485760

Cause: the first line means the batch body is larger than the endpoint, or a proxy in front of it, accepts. The second means one single event is larger than max_bytes on its own, so it can never be batched. max_bytes defaults to 10 MB, which matches the hard limit of AWS API Gateway but is ten times the default client_max_body_size of nginx. The cap is an estimate over the raw messages. It does not account for JSON escaping or form encoding, so the body that leaves the host can be larger than the number you set, and form encoding in particular can multiply it.

Fix: set max_bytes with headroom below the limit the receiver enforces, and lower batch_size as well. Enable compression: true so the body on the wire is gzipped, since the cap applies to the uncompressed body. For a single oversized event, raise max_bytes or trim the event in a pipeline. Some receivers close the connection instead of answering 413. That appears as EOF or connection reset by peer rather than as a status code.

Data impact: data is lost. Both lines are rejections, so the payload is dropped after 4 deliveries.

"x509: certificate signed by unknown authority"

Sender worker 1 execute() failed for webhook: failed to send HTTP request (non-retryable): failed to send request: Post "https://logs.example.com/ingest": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: the endpoint presents a certificate the Director host does not trust. Related texts point at the same layer. x509: certificate is valid for ..., not ... is a name mismatch. x509: certificate has expired or is not yet valid is an expiry. remote error: tls: bad certificate or tls: certificate required means the server refused, or demanded, a client certificate.

Fix: set tls.status: true first. Every tls.* key is ignored while it is false, which is why adding tls.ca_name or tls.verify: false on its own appears to change nothing. Then:

• Private or internal CA: set tls.ca_name to the CA bundle. It replaces the host trust store rather than adding to it, so the bundle must carry the full chain.
• Name mismatch: set tls.server_name to the name on the certificate, or correct the host in url.
• Mutual TLS: set tls.cert_name and tls.key_name together, and tls.passphrase when the key is encrypted. Both files must be PEM, and must be inline or resolve to a path under the Director installation directory.
• As a last resort, tls.verify: false, and only on a network path you control.

http: server gave HTTP response to HTTPS client is not a certificate problem. It means the scheme in url does not match the port.

Data impact: none. The batch is redelivered until the handshake succeeds, so the queue grows until you fix it.

"dial tcp: lookup ...: no such host" or "connect: connection refused"

Sender worker 0 execute() failed for webhook: failed to send HTTP request after 0 retries: failed to send request: Post "https://logs.example.com/ingest": dial tcp: lookup logs.example.com: no such host

Cause: the Director host cannot resolve, or cannot reach, the host in url. no such host is a name that does not resolve, or a resolver the host cannot reach. connect: connection refused is a host that answers but has nothing listening on that port. i/o timeout after connect_timeout seconds is a firewall dropping the connection. The after 0 retries part counts in-process retries, which is max_retries and defaults to 0.

Fix: check the name and the port from the Director host itself, not from your workstation. Raise max_retries, or set it to -1, when the endpoint is only briefly unreachable.

When the traffic goes through a forward proxy, note that the proxy comes from HTTP_PROXY, HTTPS_PROXY and NO_PROXY in the Director service environment. There is no per-target proxy setting. Requests to localhost, loopback and link-local addresses always go direct, whatever NO_PROXY says. A proxy that is down typically reports proxyconnect in place of dial, and a proxy that wants credentials typically refuses the tunnel with a proxy authentication response. Put the credentials in the proxy URL, or add the endpoint host to NO_PROXY. A proxy failure is typically not counted as a dial failure, so max_retries does not apply to it.

Data impact: none. Both errors are retried until the host is reachable.

"received status code 301", or a redirect that drops the body

Sender worker 0 execute() failed for webhook: failed to send HTTP request (non-retryable): received status code 302 from server:

Cause: url is not the final location. The usual triggers are an http:// URL that the endpoint redirects to https://, and a path that redirects to the same path with a trailing slash. What happens next depends on follow_redirects:

• At the default of true the redirect is followed. A 301, 302 or 303 is typically re-issued as a GET without the body, so the endpoint at the new location receives no events, answers 200, and the batch is counted as delivered. A 307 or 308 typically keeps both the method and the body.
• At false the redirect surfaces as the status code above instead.

Fix: put the final location in url. Issue one request by hand against the configured URL and follow the Location header until it stops changing. Setting follow_redirects: false while you investigate turns a silent redirect into a visible error.

Data impact: at the default, events are lost silently. They are counted as delivered while the endpoint received an empty request. With follow_redirects: false nothing is lost, and the batch is retried until url is correct.

"net/http: timeout awaiting response headers" or "context deadline exceeded"

Sender worker 3 execute() failed for webhook: failed to send HTTP request (non-retryable): failed to send request: Post "https://logs.example.com/ingest": net/http: timeout awaiting response headers

Cause: the request left the host but no answer arrived in time. socket_timeout bounds the wait for the response headers and defaults to 10 seconds. timeout bounds the whole request, defaults to 60 seconds, and produces context deadline exceeded (Client.Timeout exceeded while awaiting headers). A large batch, a slow endpoint and a slow proxy all produce these.

Fix: raise socket_timeout and timeout to match what the endpoint needs. Lower batch_size and max_bytes so each request is smaller, and enable compression: true.

Data impact: no loss, but duplicates are possible. The body was already sent, so the request is not retried in the worker. The payload returns to the queue and is redelivered. If the endpoint had already stored the batch before the timeout, it receives those events twice. The same applies to EOF and connection reset by peer in mid-request.

"received status code 429", or a 5xx from the endpoint

Sender worker 0 execute() failed for webhook: failed to send HTTP request (non-retryable): received status code 429 from server: {"error":"slow down"}

Cause: the endpoint is throttling the target (429), or it is down or overloaded (500, 502, 503, 504). Both come from the receiver.

Fix: for 429, raise batch_size so the same volume needs fewer requests, and lower the worker count on the target. For a 5xx, fix the receiver or the load balancer in front of it.

Data impact: none. The payload is redelivered until the endpoint accepts it. A 5xx returned after the receiver already stored the batch produces duplicates there.

The target looks healthy but nothing arrives

Check these in order.

1. debug.dont_send_logs is enabled. Events are processed by the pipeline and then checkpointed as delivered without ever being sent. With debug.status: true as well, start-up logs one line and nothing else: Log sending is disabled for this target (webhook). Logs will be processed by the pipeline but will not be sent to the target. With debug.status: false there is no line at all. The target's out counter stays still, because it only advances after a 2xx.

2. A redirect is swallowing the body. See the redirect entry above. This is the one failure that counts events as delivered while the endpoint receives nothing.

3. method is GET or HEAD. Both pass validation and the body is still sent, but most servers ignore a body on those methods and answer 200, which counts as delivered. Use POST, PUT or PATCH.

4. The endpoint answers 2xx and reports its own error in the body. Any 2xx counts as a delivery, and the response body is not inspected. Webhooks that answer 200 with a failure field typically fall in this group. Check the receiver's own logs, and configure it to answer 4xx or 5xx where it can.

5. tls.* keys are set while tls.status is false. They are all ignored, so a certificate fix appears to do nothing. Set tls.status: true.

6. A TLS version string is not recognized. Only tls1.0, tls1.1, tls1.2 and tls1.3 are accepted. A spelling such as TLSv1.2 falls back to the default without a warning.