Splunk
Synopsis
Creates a Splunk HTTP Event Collector (HEC) target that sends events to one or more Splunk instances. Supports batching, compression, field normalization, and automatic load balancing across multiple endpoints.
Schema
- name: <string>
description: <string>
type: splunk
pipelines: <pipeline[]>
status: <boolean>
properties:
endpoints:
- endpoint: <string>
auth_type: <string>
token: <string>
secret: <string>
index: <string>
source_type: <string>
source: <string>
batch_size: <numeric>
max_bytes: <numeric>
timeout: <numeric>
tcp_routing: <boolean>
use_compression: <boolean>
insecure_skip_verify: <boolean>
ca_name: <string>
server_name: <string>
cert_name: <string>
key_name: <string>
passphrase: <string>
min_tls_version: <string>
max_tls_version: <string>
field_format: <string>
debug:
status: <boolean>
dont_send_logs: <boolean>
Configuration
The following are the fields used to define the target:
| Field | Required | Default | Description |
|---|---|---|---|
name | Y | Target name | |
description | N | - | Optional description |
type | Y | Must be splunk | |
pipelines | N | - | Optional post-processor pipelines |
status | N | true | Enable/disable the target |
Endpoint
| Field | Required | Default | Description |
|---|---|---|---|
endpoint | Y | - | Splunk HEC endpoint URL |
auth_type | N | token | Authentication type: token or secret |
token | Y* | - | HEC token when using token auth |
secret | Y* | - | Bearer token when using secret auth |
* = token is required when auth_type is token; secret is required when auth_type is secret.
Event
| Field | Required | Default | Description |
|---|---|---|---|
index | N | - | Default Splunk index |
source_type | N | - | Default sourcetype for events |
source | N | - | Default source for events |
batch_size | N | 10000 | Number of events to batch before sending |
max_bytes | N | 4194304 (4 MB) | Maximum size of a single request body or batch, in bytes. Set it here, at the root of properties. An entry inside endpoints is ignored |
timeout | N | 30 | Time limit for the whole request in seconds, covering the connection, the upload of the batch, and the wait for the response |
max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:
- An explicit
max_bytes: 0IS honoured and removes the ceiling on targets that have no ceiling of their own. The value is read withInt64, which returns what you wrote — unlikemax_size, which is read withMustInt64and silently substitutes its default for a zero. Where a target has a fixed ceiling of its own, its page says so and a zero is clamped to that ceiling instead. - A negative value is rejected at configuration time with max_bytes must not be negative.
The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.
Connection
| Field | Required | Default | Description |
|---|---|---|---|
tcp_routing | N | false | Enable TCP routing header |
use_compression | N | true | Enable gzip compression |
field_format | N | - | Data normalization format. See applicable Normalization section |
TLS
TLS is engaged by the endpoint URL scheme: an https:// endpoint enables it, an http:// endpoint does not. There is no status field. The fields below are top-level, at the root of properties.
The web interface exposes only insecure_skip_verify for these targets. Every other field below — the CA bundle, the client certificate and key, the SNI override and the version bounds — has to be set in the YAML configuration or through the API; there is no form control for it.
| Field | Required | Default | Description |
|---|---|---|---|
insecure_skip_verify | N | false | Skip server certificate verification. Use only for testing. |
ca_name | N | - | CA bundle used to verify the server certificate. When unset, the host trust store is used; when set, it replaces the host trust store rather than adding to it. |
server_name | N | - | SNI hostname override for the TLS handshake |
cert_name | N* | - | Client certificate for mutual TLS |
key_name | N* | - | Client private key for mutual TLS |
passphrase | N | - | Passphrase for an encrypted private key |
min_tls_version | N | tls1.2 | Minimum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3) |
max_tls_version | N | tls1.3 | Maximum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3) |
* = Mutual TLS requires both cert_name and key_name. Setting only one is a configuration error.
A nested tls: block is also accepted, and a non-empty one replaces the flat shape rather than merging with it: the sender reads the block and ignores every flat key, including options the block leaves out. Use one shape or the other, never a mixture.
insecure_skip_verify is the single exception. It is read from the nested block when present and otherwise falls back to the root, so a target carrying a hand-written tls: block keeps the skip-verify its form wrote. Only an absent nested value falls through — a nested false is an explicit statement and overrides a root true.
There is no tls.status: TLS is off when the whole block is absent, and a status key inside the block is not read.
TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:
- File name — resolved relative to the service root directory. Nested paths such as
certs/prod/server.pemare supported. - Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
- Inline PEM content — used verbatim when the value contains
-----BEGIN. - Environment variable —
${ENV_VAR}. - Vault reference —
$secret{id=...}or$secret{store=...,ref=...}.
Scheduling
See Scheduling and Pool Behavior for interval and cron fields shared by all targets.
Debug Options
| Field | Required | Default | Description |
|---|---|---|---|
debug.status | N | false | Enable debug logging |
debug.dont_send_logs | N | false | Process logs but don't send to target (testing) |
Details
The Splunk HEC target sends log data to Splunk using the HTTP Event Collector (HEC) protocol. It supports multiple authentication methods, batching, compression, and automatic load balancing between endpoints.
The crowdstrike target type shares this implementation and uses the identical YAML schema. See CrowdStrike Falcon Next-Gen SIEM.
Ensure your HEC tokens have the appropriate permissions and indexes enabled in Splunk. Invalid tokens or insufficient permissions will result in ingestion failures.
Events are automatically batched and compressed by default for optimal performance. When multiple endpoints are configured, the target randomly selects an endpoint for each batch to distribute load evenly across all available Splunk instances.
Setting insecure_skip_verify to true is not recommended for production environments.
HEC Modes: JSON and RAW
The target supports both JSON and RAW HEC ingestion modes. The mode is determined by the endpoint URL path, not by a separate configuration field.
An endpoint URL ending in /services/collector/event (or the default /services/collector) uses JSON mode, where each event is sent as a structured JSON object with metadata fields (index, sourcetype, source) embedded in the payload.
An endpoint URL ending in /services/collector/raw uses RAW mode, where events are sent as plain text with metadata passed as query parameters. RAW mode is useful for forwarding pre-formatted log lines that should not be wrapped in JSON structure. When the incoming pipeline data contains a _raw field, the target extracts its value and sends it as the raw event body.
Named Streams
Endpoint URLs can include a query parameter to define a named stream. For example, https://hec.example.com:8088/services/collector/raw?paloalto creates a stream named paloalto using RAW mode, while https://hec.example.com:8088/services/collector/event?winevent creates a stream named winevent using JSON mode.
Named streams allow routing different event types through separate HEC paths within a single target configuration. A pipeline selects the stream for an event by setting index or table.
Events carrying neither are delivered to the endpoint configured without a query parameter, which acts as the catch-all stream. An event naming a stream that no endpoint defines also falls back to the catch-all; if no catch-all exists, the event cannot be delivered and is retried until one is configured.
All endpoints sharing a stream name must use the same mode. Mixing a /raw endpoint and an /event endpoint under one stream is a configuration error and the target fails to initialize.
Load Balancing and Failover
When multiple endpoints are configured, the target uses randomized load balancing. For each batch:
- Endpoints are randomly shuffled
- The batch is sent to the first endpoint
- If it fails, the next endpoint in the shuffled list is tried
- This continues until successful delivery or all endpoints fail
If only some endpoints fail but delivery eventually succeeds, the batch is cleared and a partial error is logged. If all endpoints fail, the batch is discarded from memory and a complete failure error is returned. The payload stays queued and is delivered again once the target recovers.
Dynamic Routing
Pipeline processors can override the sourcetype and select the delivery stream on a per-event basis:
- Setting
schemaorsourcein a pipeline overrides the sourcetype for that event. When both are set,schemawins. - Setting
indexortablein a pipeline selects the named stream the event is delivered to, as described above. It does not change theindexvalue embedded in the JSON payload.
The index and source values sent to Splunk are always taken from the target's own properties and cannot be overridden per event.
Example pipeline configuration:
pipelines:
- name: route_by_severity
processors:
- set:
field: schema
value: "app:error"
if: "severity == 'error'"
- set:
field: index
value: "critical"
if: "severity == 'critical'"
Events matching the second condition are delivered to the stream named critical, which requires an endpoint configured with ?critical. Events that match no stream fall back to the endpoint configured without a query parameter.
Compression
Compression is enabled by default and uses gzip to reduce network bandwidth. This adds minimal CPU overhead but can significantly improve throughput for high-volume scenarios. Disable compression only if you have bandwidth to spare and want to reduce CPU usage.
Field Normalization
Field normalization helps standardize log data before sending it to Splunk, ensuring consistent data formats that can be easily correlated:
cim- Common Information Model
Normalization is applied before batching and sending to Splunk.
Examples
Basic
Send events to a single HEC endpoint... | |
Load Balanced
Configure load balancing and failover across multiple endpoints... | |
High-Volume
Configure for high throughput with larger batches and extended timeout... | |
With Field Normalization
Using CIM field normalization for better Splunk integration... | |
Secure
Using secret-based auth with TLS verification and custom source... | |
RAW Mode
Send raw log lines using HEC RAW mode with a named stream... | |
No Compression
Disable compression to reduce CPU overhead when bandwidth is not a concern... | |
Troubleshooting
This section covers the errors you are most likely to see with the splunk target, what causes each one, and how to fix it. The splunksecurity and crowdstrike targets deliver through the same HTTP Event Collector protocol, so everything below applies to them as well. Their own pages cover what is specific to each.
Where to look:
- Director logs. Target errors are tagged with the target name and carry
"Section":"SenderPool". The part afterReason:or after the last colon is the actual cause. When the collector answered, its status code and its reply are quoted at the end of the line, afterreceived status code. - The target's connection status in the web interface. It shows the same reason as the log line, prefixed with
connection failed for <target name>:.
Starting the target does not contact the collector. A green connection status therefore only means the configuration was accepted: a wrong token, a closed port or a disallowed index is first reported when the first batch is sent.
See Target Delivery Errors for how Director logs and retries target failures.
Which setting is wrong?
Match the status code in the log line against this table first. The reply quoted after the code tells you which setting to change.
| Response | Setting at fault | What to change |
|---|---|---|
401 with Token is required | token is empty, or the authorization header never reached the collector | Set endpoints[].token. If it is filled in, check whether a proxy or load balancer in front of the collector strips the header |
403 with Invalid token, or typically Token disabled | The token was rotated, belongs to another Splunk stack, or is disabled | Paste the current token and enable it under |
400 typically reporting Incorrect index, code 7 | index, or the list of indexes the token may write to | Create the index, or add it to the token's allowed indexes. This case loses data |
400 with Invalid data format or Event field is required | The endpoint URL path does not match the body being sent, or compression was stripped in transit | Check the path against the HEC modes described above, and try use_compression: false when a proxy sits in the path. This case loses data |
400 typically reporting Data channel is missing, code 10 | Indexer acknowledgment is enabled on the token | Disable acknowledgment on the token. Director never confirms acknowledgment ids. This case loses data |
404 with Not found, or an HTML page | The path or the port in endpoint | Use /services/collector, /services/collector/event or /services/collector/raw on the collector port, not the web or management port |
413 with Data channel is full | max_bytes or batch_size is larger than the receiver accepts | Lower either one. This case loses data |
422 | The receiver refused the content. Typical of a collector-compatible gateway rather than Splunk itself | Read the reply quoted in the log line. This case loses data |
408 with Timeout, or 429 with Too many requests | Nothing in the target is wrong. The collector is refusing work it cannot take right now | Lower batch_size, add endpoints, or raise the ingest quota |
500 or 503 with Server is busy | Nothing in the target is wrong | Check the health of the Splunk indexers and their queues |
A 400, 413 or 422 is the only group treated as a permanent rejection, and only when every endpoint answers that way. If one endpoint rejects the batch while another is merely unreachable, the batch is retried instead of dropped.
"received status code 403 from Splunk", or status code 401
[Error] [director] [target-<target id>] [basic_splunk] Sender worker 3 execute() failed for basic_splunk: target broken: failed to finalize target cache: endpoint https://splunk.example.com:8088/services/collector: received status code 403 from Splunk: {"text":"Invalid token","code":4}
all endpoints failed
Cause: the collector refused the token. 403 means the token arrived but was not accepted, because it was rotated, it belongs to another Splunk stack, or it has been disabled. 401 means no usable token arrived at all, because token is empty or because something between Director and Splunk removed the authorization header.
Fix: copy the token value from token, and confirm the token is enabled there. Check every endpoint: each one carries its own token, and load balancing hides which of them is failing until you read the endpoint URL in the log line. If token holds a ${ENV} or $secret{...} reference, an unresolved reference fails earlier and differently, as described under the configuration entry below.
A token problem is retried until you fix it, so nothing is lost. The queued data is delivered once the token works.
"record rejected by target: received status code 400 from Splunk" about the index
[Error] [director] [target-<target id>] [basic_splunk] Sender worker 2 execute() failed for basic_splunk: endpoint https://splunk.example.com:8088/services/collector: record rejected by target: received status code 400 from Splunk: ...
all endpoints failed
record rejected by target
[Error] [director] [target-<target id>] [basic_splunk] Sender worker 2 deterministic failure for basic_splunk after 4 attempts — dropping (giving up): ...
Cause: the reply typically reports Incorrect index with "code":7. The value in index names an index that does not exist, or one the token is not allowed to write to. In JSON mode the index travels inside every event, in RAW mode as a query parameter, and both are refused the same way.
Fix: create the index in Splunk, or open the token and add the index to its allowed indexes. Leaving index empty makes the collector use the token's own default index, which is the quickest way to confirm that the index is the problem.
This is the one failure on this page that loses data. A rejected batch is delivered four times and then dropped, and the dropping (giving up) line is your only record of it. A token or network problem, by contrast, is retried until you fix it and loses nothing. Treat a rejection as an incident with a deadline, not as something to look at tomorrow.
"x509: certificate signed by unknown authority"
[Error] [director] [target-<target id>] [secure_splunk] Sender worker 1 execute() failed for secure_splunk: target broken: failed to finalize target cache: endpoint https://splunk.example.com:8088/services/collector: failed to send request: Post "https://splunk.example.com:8088/services/collector": tls: failed to verify certificate: x509: certificate signed by unknown authority
all endpoints failed
Cause: the certificate the collector presents is not trusted by the Director host. A fresh Splunk installation serves a self-signed certificate on the collector port until you replace it, and a certificate from your own internal authority has the same effect.
Fix: export the issuing certificate in PEM form, place it where Director can read it, and set ca_name to it. ca_name replaces the host trust store instead of adding to it, so that file has to contain every authority this target needs. Two related messages have their own fix:
certificate is valid forone namenotanother means the host inendpointis not on the certificate. Setserver_nameto a name the certificate carries, or reissue the certificate for the name you use.certificate has expired or is not yet validmeans the certificate, or the clock on either host, needs attention.
For a laboratory you can set insecure_skip_verify: true instead, which turns verification off completely. Retried until fixed, so nothing is lost.
"no such host", "connection refused", or "received status code 404 from Splunk"
[Error] [director] [target-<target id>] [basic_splunk] Sender worker 1 execute() failed for basic_splunk: target broken: failed to finalize target cache: endpoint https://splunk.example.com:8088/services/collector: failed to send request: Post "https://splunk.example.com:8088/services/collector": dial tcp: lookup splunk.example.com: no such host
all endpoints failed
Cause: the three parts of endpoint fail differently, and the text after failed to send request: says which part to look at. That text comes from the operating system, so the exact wording varies between a Linux and a Windows host.
- A
no such hostmessage is the hostname. It is a typo, or a name the Director host cannot resolve. - A
connection refusedori/o timeoutmessage is the port. The collector is not listening, or a firewall drops the connection. Splunk Enterprise listens on8088by default, and a Splunk Cloud stack typically uses443with a hostname issued for the stack. - A
404is the path, and the collector is answering. Only/services/collector,/services/collector/eventand/services/collector/raware accepted, and they have to be on the collector port rather than the web interface or management port. - A
malformed HTTP responseorserver gave HTTP response to HTTPS clientmessage is the scheme. Pointinghttps://at a plaintext port, orhttp://at a TLS port, produces one of the two. - A
context deadline exceededmessage means the collector took the connection but did not answer withintimeoutseconds. Raisetimeoutfor a slow indexer, or lowerbatch_sizeso each request is smaller.
Fix: correct the failing part of the URL, then confirm the route from the Director host itself. If the traffic goes through a proxy, HTTP_PROXY, HTTPS_PROXY and NO_PROXY of the Director service decide it, and a proxy that cannot be reached appears as a proxyconnect tcp: message. Loopback and link-local addresses always bypass the proxy. All of these are retried until fixed, so nothing is lost.
"record size ... exceeds configured max_bytes", or status code 413
[Error] [director] [target-<target id>] [perf_splunk] Sender worker 4 execute() failed for perf_splunk: record rejected by target: record size 6291616 exceeds configured max_bytes 4194304
Cause: one event on its own is larger than max_bytes, which defaults to 4194304 bytes. The size is estimated before the event is buffered, and in JSON mode the estimate allows for the envelope around the message, so an event slightly under the ceiling can still be counted over it. A 413 with Data channel is full is the same problem one step further out: the whole request body was larger than the receiver accepts.
Fix: for an oversized event, raise max_bytes, set it to 0 to remove the ceiling, or trim the event in a pipeline. For a 413, lower max_bytes and batch_size instead, because the limit being hit belongs to the receiver. Set max_bytes at the root of properties; an entry inside endpoints has no effect. The ceiling applies to the uncompressed body, so use_compression does not change it.
Both are rejections, so the batch is dropped after four deliveries. Events queued ahead of the oversized one are delivered first.
"received status code 429 from Splunk"
[Error] [director] [target-<target id>] [perf_splunk] Sender worker 0 Finalize failed on flush for target "perf_splunk": endpoint https://splunk.example.com:8088/services/collector: received status code 429 from Splunk: {"text":"Too many requests"}
all endpoints failed
Cause: the collector is refusing work it cannot take at the moment. 429 is a rate or quota limit, usually on a Splunk Cloud stack or at a gateway in front of the collector. 503 with {"text":"Server is busy","code":9} and 408 with {"text":"Timeout"} mean the indexers are saturated rather than rate limited, and are handled the same way.
Fix: lower batch_size so each request is smaller, spread the load over more endpoints, or raise the quota on the Splunk side. The batch is retried at a steady pace until it is accepted, so nothing is lost, but expect the queue to grow for as long as the limit holds.
Events arrive in Splunk but the message is unreadable
Symptom: events are indexed with the expected time, index and sourcetype, the target reports success and its delivered counters climb, but the event body is one long run of letters and digits instead of the log line.
Cause: on a JSON endpoint the message is sent base64-encoded, so the collector stores and indexes the encoded form. Searches match the encoded string rather than the original text. Nothing is logged, because the collector accepted the request and answered 200.
Fix: deliver through a RAW endpoint, which writes the log line as it stands. Change the path in endpoint from /services/collector or /services/collector/event to /services/collector/raw, keeping any stream query parameter on the end:
endpoints:
- endpoint: "https://splunk.example.com:8088/services/collector/raw"
auth_type: token
token: "YOUR-HEC-TOKEN"
Your routing is unaffected. In RAW mode index, source_type and source travel as query parameters instead of inside the payload, and an event carrying a _raw field is sent as that field's value. All endpoints of one stream have to use the same mode, so change them together.
"target not initialized" naming a stream
[Error] [director] [target-<target id>] [raw_splunk] Sender worker 2 execute() failed for raw_splunk: target broken: target not initialized (firewall)
Cause: a pipeline set index or table to firewall, which selects the named stream firewall, but no endpoint URL ends in ?firewall and no endpoint is configured without a query parameter to act as the catch-all. The variant target not initialized (no default stream) is the mirror image: the event named no stream, and every endpoint is a named one.
Fix: add an endpoint whose URL ends in ?firewall, add one endpoint with no query parameter as the catch-all, or stop setting index in the pipeline. Retried until one of those is true, and the rest of the payload waits behind it, so fix it promptly.
A startup message about the same setting is stream "firewall" configured with mixed HEC modes (raw and JSON): check endpoint URLs. It means two endpoints share a stream name but only one of them ends in /raw. Give every endpoint of a stream the same path form.
"ValidateConfig failed for target ..."
The configuration was rejected before anything was sent. The check is repeated until you change the configuration.
| Reason | Fix |
|---|---|
at least one endpoint is required for splunk target | Add an entry under endpoints with an endpoint URL |
endpoint[0]: url is required | Fill in the URL of the entry at that position, counting from zero, or delete the entry |
endpoint[0] (https://splunk.example.com:8088/services/collector): token is required for token authentication | Set token, or set auth_type: secret and fill in secret |
endpoint[0] (https://splunk.example.com:8088/services/collector): secret is required for secret authentication | Set secret, or set auth_type: token and fill in token |
endpoint[0] (https://splunk.example.com:8088/services/collector): invalid auth_type "bearer" (must be 'token' or 'secret') | Use token or secret. The value is case-sensitive |
batch_size must be greater than 0, got -5 | Remove batch_size to use the default of 10000, or set a positive number |
max_bytes must not be negative, got -1 | Set max_bytes to 0 or more. 0 removes the ceiling |
A Failed to reinitialize target ... Reason: line means the configuration passed but the target could not start. The usual reasons here are a credential reference that cannot be resolved, such as failed to resolve token: credential: env variable "SPLUNK_HEC_TOKEN" is not set, and TLS material that cannot be read, such as failed to build TLS config: ca_name: ... could not be resolved. An environment variable has to exist in the environment of the Director service, not of your shell, and certificate files have to sit under the service root. The attempt counter in the message grows while the cause persists and is not a separate fault. Data waits in the queue meanwhile.
The same events are indexed more than once
Cause: two situations produce duplicates. When one target serves several named streams and one stream fails, the whole payload is delivered again, so the streams that had already succeeded receive their events a second time. And when something in front of the collector answers with a 2xx status other than 200, a gateway replying 202 for example, the delivery counts as failed and is retried even though the gateway kept the batch.
Fix: give a stream its own target when it fails independently of the others, and configure any gateway in the path to answer 200 on success. If duplicates matter downstream, deduplicate on a record identifier in Splunk.
The target is healthy but nothing arrives in Splunk
Check these in order.
debug.dont_send_logsis enabled. Events are processed by the pipeline and then discarded instead of being sent, and no error is logged. Withdebug.statusalso enabled, startup logsLog sending is disabled for this target. Turn it off.- The events went to the catch-all endpoint. An event naming a stream that no endpoint defines is delivered to the endpoint without a query parameter rather than failing, so the data is in Splunk under that endpoint's index and sourcetype. Add the missing stream endpoint.
max_bytesis written insideendpoints. It is read from the root ofpropertiesonly, so an entry inside an endpoint is ignored and the4194304byte default still applies.- Indexer acknowledgment is enabled on the token. On a RAW endpoint the collector answers
200with an acknowledgment id, and Director never asks whether that id was confirmed. An event the indexer later fails to write is still counted as delivered. Disable acknowledgment on the token. - The endpoint URL starts with
http://. TLS is chosen by the scheme, so every TLS setting,ca_nameandinsecure_skip_verifyincluded, is ignored and the token travels in the clear. Usehttps://. - A value was corrected without a message. A
min_tls_versionormax_tls_versionthat is not one oftls1.0,tls1.1,tls1.2ortls1.3falls back to the default, and abatch_sizethat is not a number becomes10000. Both leave the target running with settings you did not choose.