Amazon OpenSearch

Amazon AWS Search & Analytics

Synopsis

Creates an Amazon OpenSearch target that sends data using the Bulk API with AWS IAM authentication. Supports multiple endpoints, field normalization, customizable batch sizing, and automatic load balancing across OpenSearch nodes.

Schema

- name: <string>
  description: <string>
  type: amazonopensearch
  status: <boolean>
  pipelines: <pipeline[]>
  properties:
    version: <numeric>
    index: <string>
    max_payload_size_kb: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    insecure_skip_verify: <boolean>
    ca_name: <string>
    server_name: <string>
    cert_name: <string>
    key_name: <string>
    passphrase: <string>
    min_tls_version: <string>
    max_tls_version: <string>
    use_compression: <boolean>
    write_action: <string>
    filter_path: <string>
    pipeline: <string>
    field_format: <string>
    endpoints:
      - endpoint: <string>
        use_iam: <boolean>
        region: <string>
        key: <string>
        secret: <string>
        session: <string>
        username: <string>
        password: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following are the fields used to define the target:

Field	Required	Default	Description
name	Y	-	Target name
description	N	-	Optional description
type	Y	-	Must be amazonopensearch
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

OpenSearch

Field	Required	Default	Description
version	N	auto	Accepted but not applied. The value is stored and never read, so no version detection or version-specific behavior follows from it
index	Y	-	Default OpenSearch index name
max_payload_size_kb	N	4096	Maximum bulk request size in KB
batch_size	N	10000	Maximum number of events per batch
timeout	N	30	Connection timeout in seconds
use_compression	N	true	Enable GZIP compression
write_action	N	create	Bulk API action (index, create, update, delete)
filter_path	N	errors,items.*.error,items.*._index,items.*.status	Response filter path
pipeline	N	-	Ingest pipeline name
field_format	N	-	Data normalization format. See applicable Normalization section

Endpoint

Field	Required	Default	Description
endpoint	Y	-	OpenSearch domain URL (automatically appends /_bulk if not present)
use_iam	N	true	Use AWS IAM authentication (recommended)
region	Y*	-	AWS region for IAM authentication
key	N*	-	AWS access key ID
secret	N*	-	AWS secret access key
session	N	-	AWS session token for temporary credentials
username	N	-	Basic auth username (alternative to IAM)
password	N	-	Basic auth password (alternative to IAM)

* = Required when use_iam is true. If key and secret are not provided, the target will use the default AWS credential chain (environment variables, IAM role, etc.).

TLS

TLS is engaged by the endpoint URL scheme: an https:// endpoint enables it, an http:// endpoint does not. There is no status field. The fields below are top-level, at the root of properties.

The web interface exposes only insecure_skip_verify for these targets. Every other field below — the CA bundle, the client certificate and key, the SNI override and the version bounds — has to be set in the YAML configuration or through the API; there is no form control for it.

Field	Required	Default	Description
insecure_skip_verify	N	false	Skip server certificate verification. Use only for testing.
ca_name	N	-	CA bundle used to verify the server certificate. When unset, the host trust store is used; when set, it replaces the host trust store rather than adding to it.
server_name	N	-	SNI hostname override for the TLS handshake
cert_name	N*	-	Client certificate for mutual TLS
key_name	N*	-	Client private key for mutual TLS
passphrase	N	-	Passphrase for an encrypted private key
min_tls_version	N	tls1.2	Minimum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3)
max_tls_version	N	tls1.3	Maximum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3)

* = Mutual TLS requires both cert_name and key_name. Setting only one is a configuration error.

note

A nested tls: block is also accepted, and a non-empty one replaces the flat shape rather than merging with it: the sender reads the block and ignores every flat key, including options the block leaves out. Use one shape or the other, never a mixture.

insecure_skip_verify is the single exception. It is read from the nested block when present and otherwise falls back to the root, so a target carrying a hand-written tls: block keeps the skip-verify its form wrote. Only an absent nested value falls through — a nested false is an explicit statement and overrides a root true.

There is no tls.status: TLS is off when the whole block is absent, and a status key inside the block is not read.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The target supports AWS IAM authentication using Signature V4, multiple endpoints, compression, and ingest pipelines. Data is batched for efficient delivery and can be automatically routed to different indices.

URLs are automatically appended with /_bulk if the suffix is not present. Events are batched until either the batch size or payload size limit is reached.

For load balancing, events are sent to randomly selected endpoints. If an endpoint fails, the next endpoint in the randomized list is tried until successful delivery or all endpoints fail.

Each event is automatically enriched with a timestamp in RFC3339 format based on the log's epoch time. You can route events to different indices by setting the index field in a pipeline processor.

warning

Long timeout values may lead to connection pooling issues and increased resource consumption.

warning

Setting max_payload_size_kb too high might cause memory pressure and can exceed OpenSearch's http.max_content_length setting (default 100MB).

AWS IAM Authentication

Amazon OpenSearch Service supports two authentication methods:

IAM Authentication (recommended)

• Uses AWS Signature V4 to sign HTTP requests
• No passwords stored in configuration
• Integrates with AWS IAM policies and roles
• Supports temporary credentials from STS
• Default authentication method for this target

Basic Authentication

• Traditional username and password
• Requires fine-grained access control enabled on the domain
• Less secure than IAM authentication

When use_iam is true, the target signs all HTTP requests with AWS Signature V4. The signing process uses the service name es as required by Amazon OpenSearch Service.

AWS Credential Chain

If key and secret are not provided in the endpoint configuration, the target uses the AWS SDK default credential chain:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
2. Shared credentials file (~/.aws/credentials)
3. IAM role for EC2 instances
4. IAM role for ECS tasks
5. IAM role for Lambda functions

This lets the target run on AWS infrastructure without any credentials in the configuration file.

Load Balancing and Failover

When multiple endpoints are configured, the target uses randomized load balancing. For each batch:

1. Endpoints are randomly shuffled
2. The batch is sent to the first endpoint
3. If it fails, the next endpoint in the shuffled list is tried
4. This continues until successful delivery or all endpoints fail

If only some endpoints fail but delivery eventually succeeds, the batch is cleared and a partial error is logged. If all endpoints fail, the batch is retained for retry and a complete failure error is returned.

JSON Message Handling

Messages that are already in JSON format take a different path from plain lines:

• A message is treated as a structured JSON document when it is ECS-normalized, or when its text contains @timestamp anywhere at all, even in a plain sentence
• Such a message is parsed and sent as-is to OpenSearch
• If parsing fails, the record is not sent as plain text. It fails with failed to parse JSON message, and it is retried until the pipeline stops producing it
• Every other message is wrapped into a JSON document automatically, with a timestamp taken from the log's epoch time

This lets you send both structured and unstructured logs through the same target, as long as anything that looks like JSON to the target really is a JSON object. See the Elasticsearch troubleshooting section for what that error looks like and how to clear it.

Dynamic Index Routing

Route events to different indices using pipeline processors by setting the index field:

pipelines:
  - name: route_by_type
    processors:
      - set:
          field: index
          value: "error-logs"
          if: "level == 'error'"
      - set:
          field: index
          value: "metrics"
          if: "type == 'metric'"

This allows flexible routing without creating multiple target configurations.

Bulk API Error Handling

The target parses the bulk API response to detect individual document errors:

• Uses filter_path to reduce response size and focus on error details
• Extracts error type, reason, and HTTP status for failed documents
• Returns detailed error messages indicating which documents failed and why

Common errors include:

• Document version conflicts (for create action)
• Mapping errors (field type mismatches)
• Index not found or closed
• Pipeline failures (when using ingest pipelines)

Write Actions

The write_action field determines how documents are indexed:

• create (default): Only index if document doesn't exist. Fails on duplicates.
• index: Index or replace existing document. Always succeeds unless there's a system error.
• update: Update existing document. Fails if document doesn't exist.
• delete: Remove document. Use carefully.

Response Filtering

The filter_path parameter filters the bulk API response to reduce network overhead:

• errors: Boolean indicating if any operations failed
• items.*.error: Error details for failed operations
• items.*._index: Index name for each operation
• items.*.status: HTTP status code for each operation

For high-volume scenarios, this filtering significantly reduces response size and parsing overhead.

Field Normalization

The field_format property allows normalizing log data to standard formats:

• ecs - Elastic Common Schema

Field normalization is applied before the logs are sent to OpenSearch, ensuring consistent indexing and search capabilities. ECS normalization maps common fields to OpenSearch's standard schema for improved compatibility with OpenSearch Dashboards and detection rules.

Compression

Compression is enabled by default and uses gzip to reduce network bandwidth. This adds minimal CPU overhead but can significantly improve throughput for high-volume scenarios. Disable compression only if you have bandwidth to spare and want to reduce CPU usage.

Examples

Basic with IAM

Simple OpenSearch output with IAM authentication...

targets: - name: opensearch_output type: amazonopensearch properties: index: "logs-%Y.%m.%d" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1" key: "AKIAIOSFODNN7EXAMPLE" secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

IAM Role

Using IAM role credentials (no explicit keys needed)...

targets: - name: iam_role_opensearch type: amazonopensearch properties: index: "application-logs" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1"

note

When using IAM role authentication on EC2, ECS, or Lambda, credentials are automatically retrieved from the instance metadata service or task role.

Temporary Credentials

Using temporary STS credentials with session token...

targets: - name: temp_creds_opensearch type: amazonopensearch properties: index: "temp-logs" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1" key: "ASIATEMP1234567890AB" secret: "tempSecretKeyExample1234567890" session: "FwoGZXIvYXdzEBYaDH...session-token"

Basic Authentication

Using basic authentication instead of IAM...

targets: - name: basic_auth_opensearch type: amazonopensearch properties: index: "secure-logs" use_compression: true endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" use_iam: false username: "admin" password: "MyPassword123"

Ingest Pipeline

Send data through an ingest pipeline for server-side processing...

targets: - name: pipeline_opensearch type: amazonopensearch properties: index: "processed-logs" pipeline: "log-processor" write_action: "create" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1"

High-Volume

Optimized for high-volume data ingestion with load balancing...

targets: - name: highvol_opensearch type: amazonopensearch properties: index: "metrics" batch_size: 20000 max_payload_size_kb: 8192 use_compression: true timeout: 60 endpoints: - endpoint: "https://search-domain1.us-east-1.es.amazonaws.com" region: "us-east-1" - endpoint: "https://search-domain2.us-east-1.es.amazonaws.com" region: "us-east-1"

Field Normalization

Using ECS field normalization for enhanced compatibility...

targets: - name: ecs_opensearch type: amazonopensearch properties: index: "normalized-logs" field_format: "ecs" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1"

Index Action

Using index action to allow document updates and overwrites...

targets: - name: index_opensearch type: amazonopensearch properties: index: "application-logs" write_action: "index" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1"

Minimal Response

Optimize for minimal response size by filtering to only errors...

targets: - name: minimal_opensearch type: amazonopensearch properties: index: "logs" filter_path: "errors" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1"

Cross-Region

Configuration with OpenSearch domains in different regions...

targets: - name: multiregion_opensearch type: amazonopensearch properties: index: "distributed-logs" endpoints: - endpoint: "https://search-domain.us-east-1.es.amazonaws.com" region: "us-east-1" - endpoint: "https://search-domain.eu-west-1.es.amazonaws.com" region: "eu-west-1"

Performance Tuning

Batch Size vs Payload Size

Events are batched until either limit is reached:

• batch_size: Number of events per batch
• max_payload_size_kb: Total size in kilobytes

Tune these based on your average event size:

• Small events (<1KB): Increase batch_size, keep default max_payload_size_kb
• Large events (>10KB): Keep default batch_size, increase max_payload_size_kb
• Mixed sizes: Monitor both limits and adjust based on actual batch sizes

Timeout

Setting appropriate timeouts helps balance reliability and performance:

• Short timeouts (10-30s): Fail fast, better for real-time scenarios
• Long timeouts (60s+): More tolerant of network issues, but may cause connection pooling problems

Compression

Enable compression (default) for high-volume scenarios to reduce network bandwidth. Disable only if CPU is constrained and network bandwidth is abundant.

Filter Path

The default filter_path provides detailed error information while minimizing response size. For even better performance in high-volume scenarios with low error rates, use filter_path: "errors" to only return the error flag.

Troubleshooting

The amazonopensearch target behaves the same way as the elastic target in everything but authentication. It builds the same bulk requests, reports the same domain errors, and retries or drops a payload on the same rules. Use the Elasticsearch troubleshooting section for the full list of errors, causes and fixes: documents the index refuses, size limits, throttling, transport and certificate failures, bulk responses that do not parse, and the cases where the target is connected but nothing arrives. This section covers only what AWS adds on top.

See Target Delivery Errors for how Director logs and retries target failures.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause. Everything up to from Elastic: is written by Director, and what follows it is the domain's own answer.
• The target's connection status in the web interface. It shows the same reason as the log line.

Nothing is checked while the target starts. Director resolves the credentials, builds one HTTP client per endpoint, and reports the target as connected without calling the domain once. The first proof that a credential or a policy works is the first batch, so permission problems always arrive as delivery errors.

What differs for Amazon OpenSearch

Requests are signed, not sent with a password. use_iam is true by default, so every batch is signed with AWS Signature Version 4 and username and password are not sent at all. region is then required on each entry under endpoints, and it has to be the region of that domain. A missing value stops the target before it starts. A wrong one is accepted locally and refused by the domain with a 403 whose body typically says the credential should be scoped to a valid region. Set use_iam: false on an entry to send a user and password instead, which needs fine-grained access control on the domain and an internal user in its own database.

Three separate things have to allow the write. The identity that signs needs the IAM action es:ESHttpPost on arn:aws:es:<region>:<account>:domain/<domain name>/*. The domain's own access policy has to allow the same principal. With fine-grained access control turned on, that identity also has to be mapped to an OpenSearch role that may write to the index. The last one is the usual surprise: an IAM policy that reads correctly still collects a 403, because the role mapping never mentions the identity. Check the mapping under Security in OpenSearch Dashboards before rewriting the policy. The target only posts to the bulk endpoint, so es:ESHttpPost is the single action it needs. It never reads domain or version information, so es:ESHttpGet and es:ESHttpPut can be left out.

Every one of those refusals arrives as received status code 403 from Elastic: followed by the domain's own answer. Read that answer to tell them apart. The wording comes from AWS, so it varies, but these are the ones you will meet.

What the body typically says	What it means	What to change
is not authorized to perform: es:ESHttpPost	No IAM policy allows the action for that identity	Grant es:ESHttpPost on the domain ARN, and allow the same principal in the domain access policy
Nothing about IAM, but the index name or indices:data/write/bulk	The request was signed and accepted, and fine-grained access control refused the write	Map the identity to an OpenSearch role that may write to that index
signature we calculated does not match	secret does not belong to key	Correct the pair. If it worked yesterday, check the host clock, because signing needs it within a few minutes of AWS time
Credential should be scoped to a valid region	region is not the domain's region	Set region to the region in the domain URL
Credential should be scoped to correct service	The endpoint is a Serverless collection, not a managed domain	Use a managed domain
security token included in the request is expired	session has expired	Replace the token, or attach a role and drop the static pair

No data is lost while any of these lasts. The payload is retried every few seconds until the write is allowed.

Set key and secret together, or neither. The pair is used only when both hold a value. With one of them empty both are ignored without a warning, and the default AWS credential chain is used instead: environment variables of the Director service, a shared profile, an ECS or EKS task role, then the EC2 instance role. Batches are then written under the host's identity rather than the one you configured, or they fail with the credential error below. Add session as well when the pair is a set of temporary credentials.

warning

OpenSearch Serverless collections are not supported. Requests are always signed for the es service, and a Serverless collection expects aoss, so every batch is refused with a 403 whose body typically complains that the credential should be scoped to the correct service. Nothing is ever indexed, the collection reports nothing, and the payload is retried every few seconds for as long as the target runs. Send to a managed OpenSearch domain instead.

"failed to sign request: failed to retrieve AWS credentials"

[Error] [director] [target-<target id>] [opensearch_output] Sender worker 1 execute() failed for opensearch_output: target broken: failed to finalize target cache: endpoint https://search-mydomain.us-east-1.es.amazonaws.com/_bulk: failed to sign request: failed to retrieve AWS credentials: failed to refresh cached credentials, no EC2 IMDS role found; all endpoints failed

Cause: the batch never reached the domain, because it could not be signed. The tail of the line comes from the AWS SDK.

• static credentials are empty means neither key nor secret reached the signer.
• failed to refresh cached credentials, followed by no EC2 IMDS role found, means the chain fell through to the instance metadata service and found nothing. The Director host has no instance role, or the metadata address cannot be reached, which is the common case inside a container.
• A sibling line, failed to sign request: failed to load AWS config:, means a shared profile named in the service environment does not exist.

Fix: give the signer one identity and make sure it belongs to the Director service, not to your own shell.

1. Set both key and secret on the endpoint, and session if they are temporary.
2. Or attach an instance role or task role to the host and leave both fields empty.
3. Or export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the service environment itself.
4. For the metadata case, allow the host to reach the instance metadata service. The proxy settings of the service do not apply to it, so a proxy is not the cause.

No data is lost. The payload stays queued and is retried every few seconds until signing succeeds.

"region is required when use_iam is enabled for endpoint"

ValidateConfig failed for target "opensearch_output": region is required when use_iam is enabled for endpoint: https://search-mydomain.us-east-1.es.amazonaws.com

Cause: use_iam is on, which is the default, and that entry under endpoints has no region. The check is per endpoint, so one entry without a region stops the whole target even when every other entry has one. The same reason also arrives as Failed to reinitialize target "opensearch_output" (attempt N). Reason: ... while the target keeps trying to start, and it repeats until the configuration changes.

Fix: set region on every entry under endpoints to the region of that domain, which is the middle label of the domain URL, us-east-1 in search-mydomain.us-east-1.es.amazonaws.com. Set use_iam: false on an entry only when you mean to send a user and password to that domain. No data is lost, but nothing is sent while the target refuses to start, and incoming records wait in the queue.

note

A credential given as ${VAR} or $secret{...} that cannot be resolved fails the same way, as failed to resolve access key:, failed to resolve secret key: or failed to resolve session token:. Export the variable for the Director service itself, or correct the store name and the reference.