Splunk S2S

Splunk Observability

Synopsis

Creates a target that sends events to a Splunk indexer over Splunk's native Splunk-to-Splunk (S2S) protocol on port 9997 — the wire protocol a Splunk universal forwarder speaks, as opposed to the HTTP Event Collector used by the splunk and splunksecurity targets. Supports weighted load balancing across an indexer tier, protocol v3/v4 framing, delivery acknowledgement, and compression.

Schema

- name: <string>
  description: <string>
  type: splunks2s
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    servers: <string[]>
    destinations:
      - address: <string>
        port: <numeric>
        weight: <numeric>
    load_balance_seconds: <numeric>
    load_balance_bytes: <numeric>
    load_balance_stats_seconds: <numeric>
    dns_resolve_seconds: <numeric>
    destination_down_seconds: <numeric>
    protocol: <string>
    index: <string>
    source: <string>
    source_type: <string>
    host: <string>
    extract_metadata: <boolean>
    ack: <boolean>
    ack_timeout: <numeric>
    compression: <boolean>
    server_name: <string>
    mgmt_port: <string>
    batch_size: <numeric>
    timeout: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    max_event_bytes: <numeric>
    nested_field_serialization: <string>
    throttle_bytes_per_second: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>
    tls:
      status: <boolean>
      verify: <boolean>
      server_name: <string>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>

Configuration

The following are the fields used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `splunks2s`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Destinations

Field	Required	Default	Description
`address`	Y*	-	Single indexer address (hostname or IP), paired with `port`
`port`	N	`9997`	Port for `address`, and the default port for any `destinations` entry that omits one
`servers`	N	-	Array of `"host:port"` strings, one indexer per entry
`destinations`	N	-	Weighted destination list. Each entry: `address`, `port` (default `9997`), `weight` (default `1`)

* = address, servers, and destinations are combinable and additive; at least one is required.

Load Balancing

Field	Required	Default	Description
`load_balance_seconds`	N	`30`	Seconds a worker holds one connection before reconsidering the destination. `0` pins the connection
`load_balance_bytes`	N	`0`	Bytes written on one connection before it is rotated. `0` rotates on time only
`load_balance_stats_seconds`	N	`300`	Window over which per-destination volume is remembered when apportioning load; half of it carries forward each rollover
`dns_resolve_seconds`	N	`600`	How often a configured hostname is re-resolved
`destination_down_seconds`	N	`30`	How long a destination that refused a connection is passed over before being retried

Protocol

Field	Required	Default	Description
`protocol`	N	`"v3"`	Wire framing: `v3` (legacy cooked, default), `v4` (compact framing), `auto` (negotiate `v4`, fall back to `v3`)
`ack`	N	`false`	Wait for indexer acknowledgement before reporting a batch delivered. Requires `protocol: v4` or `auto`
`ack_timeout`	N	`60`	Seconds to wait for acknowledgement before a batch counts as undelivered. Only meaningful with `ack: true`
`compression`	N	`false`	zlib-compress the connection
`server_name`	N	product name	Identity presented to the indexer during the handshake
`mgmt_port`	N	`"8089"`	Management port advertised during the handshake. Nothing connects to it

Event Metadata

Field	Required	Default	Description
`index`	N	-	Default Splunk index stamped on every event
`source`	N	-	Default source
`source_type`	N	-	Default sourcetype
`host`	N	-	Default host
`extract_metadata`	N	`true`	Lift `_raw`/`host`/`source`/`sourcetype`/`index`/`time` out of a Splunk-shaped JSON envelope. Explicitly configured metadata above still wins
`nested_field_serialization`	N	`"none"`	How an object/array-valued envelope field is handled: `none` (drop it) or `json` (forward its JSON text)

Batching and Delivery

Field	Required	Default	Description
`batch_size`	N	`1000`	Events buffered before a flush
`timeout`	N	`30`	Connection, handshake, and write timeout in seconds
`max_retries`	N	`3`	Retries after a failed delivery attempt. The default of `3` makes four attempts in all
`retry_delay`	N	`1`	Base for the wait between retry attempts, in seconds. Each wait is a random interval that doubles per attempt, capped at 30 seconds
`max_event_bytes`	N	`66060288` (~63 MB)	Ceiling on one event's payload. A larger event has its tail cut to this size and the shortened event is sent
`throttle_bytes_per_second`	N	`0`	Ceiling on outbound bytes per second per sender worker, measured post-compression. `0` disables it

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Enable TLS
`tls.verify`	N	`true`	Verify the server certificate
`tls.server_name`	N	-	SNI and expected server hostname. With `tls.verify` enabled, an address is verified directly when the certificate carries that address. Set this when it does not, or when the indexer expects a hostname for SNI
`tls.ca_name`	N	-	Custom CA to trust. Empty uses the OS trust store
`tls.cert_name`	N*	-	Client certificate for mutual TLS
`tls.key_name`	N*	-	Client key for mutual TLS
`tls.passphrase`	N	-	Passphrase for an encrypted client key
`tls.min_tls_version`	N	`tls1.2`	Minimum negotiated TLS version
`tls.max_tls_version`	N	`tls1.3`	Maximum negotiated TLS version

* = Conditionally required. tls.cert_name and tls.key_name must be supplied together; supplying only one fails configuration validation.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Normalization

Field	Required	Default	Description
`field_format`	N	-	Data normalization format. See applicable Normalization section

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
`debug.status`	N	`false`	Enable debug logging
`debug.dont_send_logs`	N	`false`	Process logs but don't send to target (testing)

Details

The Splunk S2S target speaks the same wire protocol a Splunk universal forwarder uses on port 9997, so an indexer accepts the data as cooked, pre-attributed events with no HEC token and no ingest-time parsing. It is a single-type target — splunks2s is its only registered type, unlike splunk/splunksecurity/crowdstrike, which share one HEC implementation.

Indexer Load Balancing

address/port configure a single destination, servers a plain list, and destinations a weighted list — the three forms are combinable and additive, so a weight can be added to one member of an existing servers list without rewriting the rest. A weight is a share of traffic, not an absolute value: keep weights within the same order of magnitude (1/5/4 works; 10/120/9 does not). Weight 0 parks a destination — configured and reachable as a last resort, but never chosen while any weighted destination is up.

A configured hostname is resolved to every address it has, and each resolved address becomes its own destination — so a single name fronting an indexer tier spreads traffic across the whole tier instead of pinning one member. TLS is still verified against the configured hostname, not the resolved address.

Two settings trigger rotation to a new destination:

load_balance_seconds is how long a worker holds one connection before reconsidering, mirroring Splunk's own autoLBFrequency default of 30. Without it, the first indexer a worker reaches keeps every batch that worker ever sends — an indexer added to the tier never receives anything, and one that was briefly down never gets its share back. 0 pins the connection.
load_balance_bytes is the volume trigger, and the one that matters for bursty input: a batch that arrives all at once fits inside any sensible time interval, so a time-only rule sends the whole burst to whichever destination the first batch picked. Measured against two indexers with time-only rotation, a 500-event burst went 500/0. Splunk carries the same pair of levers as autoLBFrequency and autoLBVolume. 0 rotates on time only.

load_balance_stats_seconds is the window over which per-destination volume is remembered when apportioning load; half of it carries forward on each rollover, so short-term imbalance decays instead of either persisting indefinitely or being forgotten between windows. dns_resolve_seconds controls how often a configured hostname is re-resolved — a resolution that fails keeps the last known address set rather than emptying it. destination_down_seconds is how long a destination that refused a connection is passed over; the cooldown is shared across every worker of the target, so one worker's discovery is not re-learned by the rest.

Protocol Versions

protocol: v3 (the default) sends legacy cooked messages, where each message carries its own _done terminator — the indexer takes the event exactly as sent, so the original timestamp is preserved and a multi-line event stays one event. v4 (or auto, which negotiates v4 and falls back to v3 if the receiver refuses it) sends the compact framing a Universal Forwarder uses, which hands event-boundary detection and timestamp extraction back to the indexer. ack: true requires protocol: v4 or auto — acknowledgement rides on v4 message IDs, so it cannot be combined with an explicitly configured v3.

Event Metadata

index, source, source_type, and host set the attribution stamped on every event; leaving them empty lets the source's own attribution through. When extract_metadata is enabled (the default), the target lifts _raw/host/source/sourcetype/index/time out of a Splunk-shaped JSON envelope, so data received from another Splunk source keeps its original attribution instead of arriving as an opaque JSON blob — any of the fields above that are explicitly set still take priority. Every other envelope member is forwarded on as an indexed field; nested_field_serialization decides what happens when one of those fields is an object or array, which has no representation in Splunk's flat, text-only indexed fields: none (the default) drops it, json forwards its JSON text.

If an event has no host attribution after metadata resolution, the target falls back to the originating device's name rather than letting Splunk misattribute it to the receiving indexer. Setting schema, source, or log_type in a pipeline overrides source_type for that event, the same last-write-wins mechanism the splunk target uses for its sourcetype override.

Compression

compression: true zlib-compresses the connection, as a forwarder's compressed = true setting does. It is worth enabling on a slow or metered link — a 211 KB batch of repetitive log text went out in 3.7 KB in testing — at the cost of CPU on both ends.

Payload Limits

max_event_bytes bounds a single event's payload before it is framed. Splunk's receiver enforces a hard ceiling near 64 MB and does not report it as an error — it resets the TCP connection instead. An event that exceeds max_event_bytes has its tail cut to that size and is sent shortened, so the batch still goes through and the record is still indexed, minus its tail. Without a ceiling, an oversized event that keeps getting retried resets the connection every time and wedges the destination in a reconnect loop that looks like a network fault.

Delivery and Retry

max_retries and retry_delay only absorb a brief blip in reaching a destination. Once retries are exhausted, the target returns the send error and the sender pool NAKs the source records for redelivery — which is what actually prevents loss, and which requires persistent_storage to be enabled on the Director.

Throttling

throttle_bytes_per_second caps outbound bytes per second on each sender worker, measured on what is actually written to the wire — so on a compressed connection the ceiling applies to the post-compression byte count. 0 disables it.

Examples

Basic

Sending events to a single indexer over the S2S protocol...

targets:
  - name: splunks2s_basic
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997

Indexer Tier with Failover

Spreading events across an indexer tier with automatic failover...

targets:
  - name: splunks2s_tier
    type: splunks2s
    properties:
      servers:
        - "idx1.example.com:9997"
        - "idx2.example.com:9997"
        - "idx3.example.com:9997"

Weighted Destinations

Favoring a larger indexer over two smaller ones with weighted destinations...

targets:
  - name: splunks2s_weighted
    type: splunks2s
    properties:
      destinations:
        - address: "idx1.example.com"
          port: 9997
          weight: 5
        - address: "idx2.example.com"
          port: 9997
          weight: 1
        - address: "idx3.example.com"
          port: 9997
          weight: 1

Acknowledged Delivery

Waiting for indexer acknowledgement on the compact v4 framing...

targets:
  - name: splunks2s_acked
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997
      protocol: auto
      ack: true
      ack_timeout: 60

Secure (TLS)

Connecting to an indexer tier over TLS, verified by hostname...

targets:
  - name: splunks2s_tls
    type: splunks2s
    properties:
      address: "indexers.example.com"
      port: 9997
      tls:
        status: true
        verify: true
        server_name: "indexers.example.com"
        ca_name: "splunk-ca.pem"
        min_tls_version: "tls1.2"

High-Volume with Compression

Compressing the connection and capping outbound bandwidth for a bursty, high-volume source...

targets:
  - name: splunks2s_highvolume
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997
      batch_size: 5000
      compression: true
      load_balance_bytes: 10485760
      throttle_bytes_per_second: 5242880

Troubleshooting

This section covers the errors you are most likely to see with the splunks2s target, what causes each one, and how to fix it.

Where to look:

Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

This target does not open a connection when it starts. The first connection to an indexer is made when the first batch is ready to send. An indexer that is down, a closed port, and a port that belongs to something else therefore never show as a failed connection: the connection status in the web interface stays positive, and the only sign of trouble is the delivery errors below. The status turns negative only for faults Director can see without the network, such as certificate material it cannot read.

Every failure on this page is retried, and this target never gives up on a batch. A permanent mistake such as the wrong port is redelivered for as long as it lasts. Nothing is dropped, but nothing moves either until you fix the cause.

What the indexer needs

This protocol carries no token and no password. What the target needs is a receiving port it can reach and, with TLS, certificates both sides accept.

Prerequisite	When it applies	Error when it is missing
A receiving input on the port, `[splunktcp://9997]` in the indexer's `inputs.conf`	Always	`connect: connection refused`
An SSL receiving input, `[splunktcp-ssl:9997]` with an `[SSL]` stanza	`tls.status: true`	`first record does not look like a TLS handshake`, or `EOF` when the mismatch runs the other way
Outbound TCP from the Director host to every indexer on the configured port, and DNS for every configured hostname	Always	`i/o timeout`, `no such host`
The indexer accepts a connection from the Director host, for example through its `acceptFrom` setting	Always	`receiver refused the connection: ...`, typically
The CA that signed the indexer certificate, in `tls.ca_name` or in the Director host's trust store	`tls.status: true` with `tls.verify: true`	`x509: certificate signed by unknown authority`
A client certificate and key in `tls.cert_name` and `tls.key_name`, signed by a CA the indexer trusts through `sslRootCAPath`	Only when the indexer sets `requireClientCert = true`	`remote error: tls: certificate required`
Read permission for the Director service account on the certificate, key, and CA files	TLS material given as file paths	`read key "...": ... permission denied`
The index named in `index` exists on the indexer	When `index` is set, or an envelope carries one	None. The indexer typically discards events for an index that does not exist

"connect: connection refused"

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 3 execute() failed for splunks2s_basic: failed to send after 3 retries: failed to connect to indexer.example.com:9997: dial tcp indexer.example.com:9997: connect: connection refused

Cause: nothing is listening on that port. On a Splunk indexer this almost always means receiving is not enabled, because an indexer does not listen on 9997 until you turn it on. The other causes are a port that is not the receiving port, and an indexer whose splunkd is stopped.

Fix: on the indexer, open Settings > Forwarding and receiving > Configure receiving and add the port, or add a [splunktcp://9997] stanza to inputs.conf. splunk btool inputs list splunktcp shows what is configured. Then check the port in address, servers, and destinations. When a tier has several destinations and all of them refuse, the message names only one of them, so check every indexer in the tier rather than the one in the log line. No data is lost: the target makes four attempts, then the payload returns to the queue and is redelivered until an indexer accepts it.

"x509: certificate signed by unknown authority"

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 1 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: tls.status is on, and the certificate the indexer presents is not signed by a CA Director trusts. A Splunk installation ships with its own self-signed certificate, so this is the normal result of pointing TLS at a default indexer while tls.ca_name is empty and the host trust store is used.

Fix: set tls.ca_name to the CA chain that signed the indexer certificate. On a default installation that is Splunk's own cacert.pem under $SPLUNK_HOME/etc/auth. Copy it where the Director service can read it, or paste it inline as PEM. Nothing is lost while Director keeps running, within the queue retention limit, and it is retried until you fix the cause.

warning

tls.verify: false also makes the error go away, and takes the protection with it. Use it only in a lab.

"certificate is valid for ..., not ..." or "doesn't contain any IP SANs"

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 1 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: failed to verify certificate: x509: certificate is valid for indexer.example.com, not indexers.example.com

Cause: the certificate is trusted, but the name being verified is not on it. tls.server_name is the name that is checked, and with no tls.server_name the configured hostname is used instead.

Fix: set tls.server_name to a name the certificate carries, or configure the destination under a name the certificate carries. The second form of the error is what a destination configured as an address gives you instead:

... failed to connect to <indexer address>:9997: tls: failed to verify certificate: x509: cannot validate certificate for <indexer address> because it doesn't contain any IP SANs

An address has no hostname to fall back on, so verification needs the certificate to carry that address as a subject alternative name. Set tls.server_name whenever a destination is an address and tls.verify is on. A configured hostname is resolved to all of its addresses and each one is dialled separately, but verification still uses the hostname, so a name-based destination does not need this.

"stream desynchronised: v3 body length 1213486160"

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 4 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:8088: splunks2s: stream desynchronised: v3 body length 1213486160

Cause: the port answered, but not with S2S. The first four bytes of the reply are read as the length of the message that follows, and a length outside the 4 byte to 64 MiB range produces this message. The number is the giveaway. 1213486160 is the four letters HTTP read as a length, so the port is serving HTTP: the HTTP Event Collector, the web interface, or the management port.

Fix: point the target at the indexer's receiving port, 9997 by default. This target cannot use an HTTP Event Collector port. If what you have is a HEC token rather than a receiving port, use the splunk target instead, which speaks HTTP to that endpoint. mgmt_port is not involved: it is only a number the target announces about itself, and changing it never fixes this error.

"first record does not look like a TLS handshake", or a handshake that ends in "EOF"

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 2 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: first record does not look like a TLS handshake

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 2 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:9997: EOF

Cause: the two sides disagree about TLS. The first message means tls.status: true was used against a plaintext receiving input. The second means the indexer accepted the connection and then closed it without answering, which usually means the opposite mismatch: the input is SSL only while tls.status is off. compression: true against a receiver that cannot read compressed messages ends the same way, because there is nothing that negotiates it.

Fix: make both ends agree. Either configure [splunktcp-ssl:9997] with an [SSL] stanza on the indexer and set tls.status: true, or leave both sides plaintext. If TLS already matches, turn compression off and try again.

"receiver refused the connection"

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 1 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:9997: receiver refused the connection: <the reply the indexer sent>

Cause: the connection was established, and the indexer answered the capability exchange with a refusal instead of an acceptance. The text after the last colon is that reply. It typically carries a cap_response=failure field with a reason= value naming the refusal, so read that value first.

Fix: the decision was made on the indexer, so its own logs hold the detail. Look in splunkd.log for TcpInputProc entries at the same time, then review the [splunktcp] settings that can refuse a sender, acceptFrom first. Nothing is lost, and every batch is retried until the indexer accepts the connection.

"i/o timeout" or "no such host" while connecting

[Error] [director] [target-<target id>] [splunks2s_tier] Sender worker 0 execute() failed for splunks2s_tier: failed to send after 3 retries: failed to connect to indexer.example.com:9997: dial tcp indexer.example.com:9997: i/o timeout

Cause: i/o timeout means the connection attempt went unanswered for the whole of timeout, 30 seconds by default, which is what a firewall that drops packets rather than refusing them looks like. The other form of this error, dial tcp: lookup indexer.example.com: no such host, means the name does not resolve on the Director host.

Fix: allow outbound TCP to every indexer on the configured port, and confirm the address. For the name, fix DNS or the hosts file, or configure addresses instead. Nothing is lost while either lasts. Note that a hostname is re-resolved every dns_resolve_seconds, and a re-resolution that fails keeps the last known addresses rather than emptying the list, so a tier that has moved keeps being dialled at its old addresses until those refuse.

"write: i/o timeout", or an acknowledgement that never arrives

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 3 execute() failed for splunks2s_basic: failed to send after 3 retries: write tcp ...->indexer.example.com:9997: write: i/o timeout

Cause: the indexer took the connection but stopped keeping up. This message means it stopped reading from the socket before timeout expired, which on an indexer points at blocked parsing or indexing queues. With ack: true the same stall arrives as waiting for acknowledgement: read tcp ...: i/o timeout, meaning the indexer accepted the bytes and did not acknowledge them within ack_timeout, 60 seconds by default.

Fix: look at the indexer's queue health in metrics.log, and at blocked-queue messages in splunkd.log, before changing anything here. Raise timeout or ack_timeout only when the link itself is slow. The batch counts as undelivered and is sent again, so events the indexer did write are indexed twice.

"connection reset by peer" or "broken pipe"

[Error] [director] [target-<target id>] [splunks2s_highvolume] Sender worker 2 execute() failed for splunks2s_highvolume: failed to send after 3 retries: write tcp ...->indexer.example.com:9997: write: connection reset by peer

Cause: the indexer closed the connection in the middle of the batch. The usual reasons are an indexer restart, a sender the indexer will not accept, and a single event above Splunk's receive ceiling near 64 MB. Splunk resets the connection for an oversized event instead of reporting one, so a raised max_event_bytes shows up here looking like a network fault.

Fix: check whether the indexer restarted, and keep max_event_bytes at or below its default. The destination is passed over for destination_down_seconds and the whole batch is retried on a fresh connection, so whatever was written before the reset may be indexed twice.

"acknowledgement was requested but the receiver refused v4"

[Error] [director] [target-<target id>] [splunks2s_acked] Sender worker 3 execute() failed for splunks2s_acked: failed to send after 3 retries: acknowledgement was requested but the receiver refused v4, so no message ids exist to acknowledge

Cause: acknowledgement rides on v4 message IDs, and the receiver did not grant v4. It either held the connection on v3, or never answered the capability exchange at all, which is treated the same way. A Splunk indexer grants v4, so in practice this means the port belongs to something that is not one.

Fix: confirm the port is a Splunk receiving input. If the receiver really is a third-party sink that speaks only v3, remove ack. Every batch fails this way until the configuration or the receiver changes, and each one is redelivered rather than dropped, so the queue grows in the meantime.

Configuration rejected before anything is sent

These arrive as ValidateConfig failed for target "splunks2s_basic": ... and are re-checked until the configuration changes. Nothing is sent in the meantime.

Reason text	Fix
`invalid server "...": expected host:port`	Every `servers` entry needs a port, and an IPv6 literal needs brackets, `[...]:9997`. Only `address` may be written without one
`address (or servers/destinations) is required for splunks2s target`	Set at least one of `address`, `servers`, and `destinations`
`destination 0: address is required`	Every `destinations` row needs an `address`. The number is the row's position in the list
`destination "...": weight must be zero or greater, got -5`	Weights cannot be negative. `0` parks a destination rather than removing it
`port must be between 1 and 65535, got 70000`	Correct the port
`invalid protocol "V4" (must be "v3", "v4" or "auto")`	The value is case-sensitive. Write it in lower case
`ack is only supported with protocol v4 or auto`	`ack: true` needs `protocol: auto` or `protocol: v4`. The default is `v3`
`invalid nested_field_serialization "flatten" (must be "none" or "json")`	Use `none` or `json`
`batch_size must be greater than 0, got -1`	Remove the setting to take the default, or set a positive value
`both tls.cert_name and tls.key_name must be provided together or omitted together`	Supply both, or neither

"failed to build TLS config"

TLS material is loaded before any connection is attempted, so these are the one family of faults that does reach the connection status: Failed to reinitialize target "splunks2s_tls" (attempt 3). Reason: failed to build TLS config: .... Retries back off from 5 seconds to once a minute and continue until the material loads. Incoming data waits in the queue, and no restart is needed once the file is in place.

Reason after `failed to build TLS config:`	Fix
`ca_name: "..." could not be resolved (env/vault token, inline PEM, or a path under the service root)`	The file is missing, unreadable, or outside the Director service root. Move it under the service root and reference it relatively, or paste the PEM inline
`ca_name: credential: env variable "..." is not set`	The `${...}` reference names a variable the Director service does not have. Export it for the service itself, not only in your own shell
`ca_name "..." contains no valid PEM certificate(s)`	The file is not PEM. Convert it, so that it starts with `-----BEGIN CERTIFICATE-----`
`client certificate: read key "...": ... permission denied`	Grant the Director service account read access to the key
`client certificate: decrypt PKCS#8 private key (wrong passphrase?): ...`	Correct `tls.passphrase`, or supply a key that is not encrypted
`client certificate: tls: private key does not match public key`	`tls.cert_name` and `tls.key_name` come from different pairs
`min TLS version "tls1.3" is greater than max TLS version "tls1.2"`	Correct the `tls.min_tls_version` and `tls.max_tls_version` range

The target is healthy but nothing arrives in the index

Nothing fails here, so there is no error to search for. Check the following in order.

debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No counter moves, nothing reaches the indexer, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well.
```
Log sending is disabled for this target (splunks2s_basic). Logs will be processed by the pipeline but will not be sent to the target.
```
The index does not exist on the indexer. Delivery succeeds because the indexer accepts the events first and decides where to put them afterwards. An index that does not exist typically means they are discarded on the indexer, with nothing reported back and nothing in the target's counters, unless the indexer is configured with a last-chance index. Create the index, or set index to one that exists. An event sent with no index at all typically lands in the indexer's default index, main.
The port is not a Splunk receiving input. A plain TCP listener that accepts the connection and never answers the capability exchange is treated as a receiver that speaks only v3. Events are written to it and counted as delivered while the receiver makes nothing of them. Each new connection also waits the whole of timeout for the answer that never comes, which shows up as a stall every load_balance_seconds. Confirm the port, and set ack: true with protocol: auto to turn the silence into the hard error above.
Delivery is not acknowledged by default. With ack off, a batch counts as delivered once it has been written to the socket. An indexer that closes, or that drops the data just after accepting it, loses that batch with no error anywhere. Set ack: true with protocol: auto for at-least-once delivery.
Fields are missing from the events that do arrive. With nested_field_serialization at its default of none, an envelope field whose value is an object or an array is dropped from the event, as are null and empty values. The event itself is delivered. Set nested_field_serialization: json to forward them as JSON text. An index, source, source_type, or host configured on the target also replaces whatever the envelope carried, silently.
One indexer in the tier is dead. When one destination refuses a connection and another accepts, the batch succeeds and nothing is logged. The failed destination is passed over for destination_down_seconds and tried again later. A destination with weight: 0 is never chosen at all while a weighted one is available. Watch the per-destination event counts in the target's Indexers summary for a member that has stopped receiving its share.
Events are shorter than they were. Events above max_event_bytes have their tails cut, and the shortened versions are delivered. One warning per batch reports it, carrying the number of events it shortened. No target counter moves.
```
[SplunkS2S] truncated file=<payload> target=splunks2s_basic events=2 limit=66060288
```

Synopsis​

Schema​

Configuration​

Destinations​

Load Balancing​

Protocol​

Event Metadata​

Batching and Delivery​

TLS​

Normalization​

Scheduling​

Debug Options​

Details​

Indexer Load Balancing​

Protocol Versions​

Event Metadata​

Compression​

Payload Limits​

Delivery and Retry​

Throttling​

Examples​

Basic​

Indexer Tier with Failover​

Weighted Destinations​

Acknowledged Delivery​

Secure (TLS)​

High-Volume with Compression​

Troubleshooting​

What the indexer needs​

"connect: connection refused"​

"x509: certificate signed by unknown authority"​

"certificate is valid for ..., not ..." or "doesn't contain any IP SANs"​

"stream desynchronised: v3 body length 1213486160"​

"first record does not look like a TLS handshake", or a handshake that ends in "EOF"​

"receiver refused the connection"​

"i/o timeout" or "no such host" while connecting​

"write: i/o timeout", or an acknowledgement that never arrives​

"connection reset by peer" or "broken pipe"​

"acknowledgement was requested but the receiver refused v4"​

Configuration rejected before anything is sent​

"failed to build TLS config"​

The target is healthy but nothing arrives in the index​

Synopsis

Schema

Configuration

Destinations

Load Balancing

Protocol

Event Metadata

Batching and Delivery

TLS

Normalization

Scheduling

Debug Options

Details

Indexer Load Balancing

Protocol Versions

Event Metadata

Compression

Payload Limits

Delivery and Retry

Throttling

Examples

Basic

Indexer Tier with Failover

Weighted Destinations

Acknowledged Delivery

Secure (TLS)

High-Volume with Compression

Troubleshooting

What the indexer needs

"connect: connection refused"

"x509: certificate signed by unknown authority"

"certificate is valid for ..., not ..." or "doesn't contain any IP SANs"

"stream desynchronised: v3 body length 1213486160"

"first record does not look like a TLS handshake", or a handshake that ends in "EOF"

"receiver refused the connection"

"i/o timeout" or "no such host" while connecting

"write: i/o timeout", or an acknowledgement that never arrives

"connection reset by peer" or "broken pipe"

"acknowledgement was requested but the receiver refused v4"

Configuration rejected before anything is sent

"failed to build TLS config"

The target is healthy but nothing arrives in the index