Amazon SQS

Amazon AWS Message Queue

Synopsis

Creates a target that writes log messages to Amazon Simple Queue Service (SQS) with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits. Amazon SQS is a fully managed message queuing service that enables decoupling and scaling of distributed systems and serverless applications.

Schema

- name: <string>
  description: <string>
  type: amazonsqs
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    queue: <string>
    queue_url: <string>
    max_events: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazonsqs
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom SQS endpoint URL (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Queue Configuration

Field	Required	Default	Description
queue	N*	-	SQS queue name (will be resolved to queue URL)
queue_url	N*	-	Direct SQS queue URL
max_events	N	10	Maximum number of events per batch (1-10)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

* = Either queue or queue_url must be specified. Using queue_url is more efficient as it skips the queue name resolution step.

note

Amazon SQS supports a maximum of 10 messages per SendMessageBatch request. The max_events parameter must be between 1 and 10.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Amazon SQS is a fully managed message queuing service that enables asynchronous communication between distributed system components. This target allows you to send log messages to SQS queues for processing by downstream applications.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM Action	Purpose
sts:GetCallerIdentity	Validate credentials at initialization
sqs:SendMessage	Send messages to the queue. This is the action that authorizes the SendMessageBatch call
sqs:GetQueueUrl	Resolve queue name to URL (only when queue is used instead of queue_url)

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "STSIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "SQSSendMessages",
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME"
    }
  ]
}

note

The sqs:GetQueueUrl action is only required when the queue parameter (queue name) is used. If queue_url (full URL) is provided directly, this action is not needed.

Queue Types

Amazon SQS offers standard queues and FIFO queues. This target sends to standard queues:

Standard Queues

• Nearly unlimited throughput
• At-least-once delivery
• Best-effort ordering
• Default and most common type

warning

FIFO queues, whose names end in .fifo, are not supported. A FIFO queue requires a message group id on every message, and this target does not set one, so messages sent to a FIFO queue are refused. Use a standard queue.

Message Properties

SQS messages have the following characteristics:

• Maximum message size: 1 MiB (1,048,576 bytes)
• Message retention: 1 minute to 14 days (default 4 days)
• Delivery delay: 0 seconds to 15 minutes
• Visibility timeout: 0 seconds to 12 hours

Batch Processing

The target accumulates messages in memory and sends them in batches using the SendMessageBatch API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10 messages per request (Amazon SQS limit).

Dead Letter Queues

SQS supports dead letter queues (DLQ) for handling messages that cannot be processed successfully. Configure DLQ settings in the AWS Console or via infrastructure as code.

Encryption

SQS supports server-side encryption using AWS KMS. Messages are encrypted at rest. All connections to SQS use HTTPS endpoints for encryption in transit.

Integration with AWS Services

SQS integrates with other AWS services:

• AWS Lambda for serverless message processing
• Amazon EC2 and ECS for consumer applications
• Amazon CloudWatch for monitoring and alarms
• AWS Step Functions for workflow orchestration
• Amazon EventBridge for event-driven architectures

Examples

Basic Configuration with Queue Name

The minimum configuration using queue name:

targets:
  - name: basic_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "application-logs"

With Queue URL

Configuration using direct queue URL (more efficient):

targets:
  - name: sqs_with_url
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/application-logs"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
  - name: iam_sqs
    type: amazonsqs
    properties:
      region: "us-east-1"
      queue: "application-logs"

note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate SQS permissions attached.

High Throughput

Configuration optimized for high-volume data:

targets:
  - name: high_volume_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/high-volume-logs"
      max_events: 10
      timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
  - name: temp_creds_sqs
    type: amazonsqs
    properties:
      key: "ASIATEMP1234567890AB"
      secret: "tempSecretKeyExample1234567890"
      session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
      region: "us-west-2"
      queue: "temporary-logs"

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "normalized-logs"
      field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
  - name: reliable_sqs
    type: amazonsqs
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "critical-logs"
      max_events: 5

Multiple Regions

Configuration for SQS queue in different region:

targets:
  - name: eu_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      queue: "eu-application-logs"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "scheduled-logs"
      max_events: 10
      interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "test-logs"
      debug:
        status: true
        dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing (e.g., LocalStack, ElasticMQ):

targets:
  - name: local_sqs
    type: amazonsqs
    properties:
      key: "test"
      secret: "test"
      region: "us-east-1"
      endpoint: "http://localhost:4566"
      queue: "local-test-queue"

Production Configuration

Configuration for production with optimal settings:

targets:
  - name: production_sqs
    type: amazonsqs
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/production-logs"
      max_events: 10
      timeout: 60
      field_format: "cim"

Troubleshooting

This section covers the errors you are most likely to see with the amazonsqs target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason:, or after the last colon, is the actual cause. See Target Delivery Errors for how Director logs and retries target failures.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

This target never reports a record as rejected, so every delivery error below is retried until you fix it rather than dropped after a set number of attempts. Data waits in the Director queue meanwhile.

Which permission is missing?

Match the error you see against this table first. Which action you need depends on how the queue is configured.

Error text	Missing IAM action	Resource it must be allowed on
failed to get queue url for ... with a 403 and an access denied code, typically AccessDenied	sqs:GetQueueUrl	The queue ARN, for example arn:aws:sqs:us-east-1:000000000000:application-logs. Needed only when queue is set, because queue_url skips the lookup
failed to send messages to sqs: ... SendMessageBatch ... with a 403 and an access denied code, typically AccessDenied	sqs:SendMessage	The queue ARN. sqs:SendMessage is the action that authorizes the SendMessageBatch call
failed to send messages to sqs: ... with KmsAccessDenied, KmsDisabled, or another code starting with Kms	kms:GenerateDataKey and kms:Decrypt	The customer-managed key the queue is encrypted with. Queues that use SQS-managed encryption need nothing extra

The minimum policy for delivery:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SQSSendMessages",
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:us-east-1:000000000000:application-logs"
    }
  ]
}

There is no sqs:SendMessageBatch IAM action. A policy that grants only that string grants nothing, and every flush fails with an access denied error. For a queue owned by another AWS account, the queue's own resource policy must also allow your principal, and you have to set queue_url, because the name lookup resolves queues in your own account only.

"either queue or queue_url is required for amazon sqs target"

ValidateConfig failed for target "production_sqs": either queue or queue_url is required for amazon sqs target

Cause: neither queue nor queue_url is set. The other validation error, max_events must be between 1 and 10, got 50, means max_events is outside the range one SendMessageBatch call allows.

Fix: set queue_url, or queue together with region. Remove max_events to use the default of 10, or set a value from 1 to 10.

The configuration is rejected, so the target never starts and nothing is sent until you change it.

What "Failed to reinitialize target ... (attempt N)" means

[Error] [director] [target-<target id>] [production_sqs] Failed to reinitialize target "production_sqs" (attempt 12). Reason: operation error STS: GetCallerIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region

The target could not start, and Director is retrying with a growing interval, so a high attempt count only means the cause in Reason: has been present for a while. Fix that cause and the next attempt picks it up. No restart is needed. Invalid Configuration: Missing Region in particular means region is empty and the Director host supplies no region of its own either, so set region on the target.

Nothing is sent while the target fails to start, and incoming data waits in the queue.

"InvalidClientTokenId" or "SignatureDoesNotMatch" at startup

Failed to reinitialize target "production_sqs" (attempt 3). Reason: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: ..., api error InvalidClientTokenId: ...

Cause: the credentials are checked against AWS before any queue call, and AWS refused them. The code names the field. Typically it is InvalidClientTokenId when the access key is unknown or has been rotated, SignatureDoesNotMatch when the secret is wrong or the two values are swapped, and ExpiredToken when temporary credentials have run out.

Fix: re-enter key and secret as a pair. For a key that starts with ASIA, also supply a matching, current session token. This check only proves that your credentials sign correctly, so a queue permission is not what is failing here.

Retried until fixed, and nothing reaches the queue in the meantime.

"no EC2 IMDS role found" when only one credential field is set

Failed to reinitialize target "production_sqs" (attempt 6). Reason: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded

Cause: the credentials you entered are used only when key and secret both resolve to a non-empty value. If either one is empty, both are ignored and the host's own credentials are used instead: environment variables, a shared profile, a container role, or the instance role. A host that is not on AWS has none of these. A ${ENV} or $secret{...} reference that resolves to an empty string counts as empty.

Fix: set both key and secret, or neither. To use a role, attach an instance or task role to the Director host and leave both fields empty.

Retried until fixed.

warning

On an EC2 or ECS host this fallback succeeds instead of failing. Delivery continues under the instance or task role rather than under the credentials you entered, so messages can go out under an identity you did not intend. There is no error to see.

"failed to resolve access key" or "failed to resolve secret key"

Failed to reinitialize target "production_sqs" (attempt 4). Reason: failed to resolve access key: credential: env variable "AWS_KEY" is not set

Cause: the token in key, secret, or session could not be resolved. The ending tells you which step failed, for example credential: env variable "AWS_KEY" is not set, credential: store "vault-prod" not found in configuration, credential: no credentials configured, or context deadline exceeded when the store does not answer within 30 seconds.

Fix: export the variable in the Director service environment, or correct the secret store definition and the reference in the field. session is only resolved when key and secret both have values.

Retried until fixed.

"failed to get queue url ... QueueDoesNotExist"

Failed to reinitialize target "production_sqs" (attempt 1). Reason: failed to get queue url for application-logs: operation error SQS: GetQueueUrl, https response error StatusCode: 400, RequestID: ..., QueueDoesNotExist: ...

Cause: no queue with that name exists in that region in your account. Queue names are case-sensitive, so Application-Logs and application-logs are different queues. A queue in another region, or in another account, is not found either.

Fix: correct queue and region, or set queue_url and drop queue. The same code on the send call instead of at startup means queue_url points at a queue that does not exist, or the queue was deleted while the target was running. A malformed URL is reported as InvalidAddress, so copy the value shown as URL on the queue's page in the Amazon SQS console rather than assembling it by hand, and do not paste the queue ARN.

Retried until fixed.

"failed to send messages to sqs" with an access denied code

Sender worker 3 execute() failed for production_sqs: target broken: failed to finalize target cache: failed to send messages to sqs: operation error SQS: SendMessageBatch, https response error StatusCode: 403, RequestID: ..., api error AccessDenied: ...

Cause: the target started, but AWS refused the send. The code is typically AccessDenied, and the message names the principal and the action it was denied. Startup can pass cleanly even so, because reading the queue URL and sending to the queue are separate permissions.

Fix: grant sqs:SendMessage on the queue ARN, using the policy above. For a cross-account queue, add your principal to the queue's resource policy as well. If the code starts with Kms, the queue is encrypted with a customer-managed key instead: grant kms:GenerateDataKey and kms:Decrypt on that key, and check that the key is enabled and not pending deletion.

Retried until fixed, so delivery resumes on its own once the policy is in place.

"failed to send 2/10 messages to sqs"

Sender worker 3 execute() failed for production_sqs: target broken: failed to finalize target cache: failed to send 2/10 messages to sqs

Cause: SQS accepted the request but rejected individual messages inside it. The count is the only detail reported. Most often the body contains characters SQS does not accept: a message body may carry only the characters XML allows, which are #x9, #xA, #xD, and the ranges #x20 to #xD7FF, #xE000 to #xFFFD, and #x10000 to #x10FFFF. Raw logs with NUL bytes, escape sequences from ANSI colour codes, or other control characters are rejected entry by entry, and nothing removes them for you. A single message above the queue's size limit and a FIFO queue fail the same way. See the two entries below.

Fix: strip control characters in a pipeline before the target, shrink oversized records, or move off a FIFO queue.

Retried until fixed. The whole batch goes out again on every retry, so the messages SQS already accepted arrive more than once while the failing ones keep failing.

"BatchRequestTooLong", or a message the queue refuses for its size

Sender worker 1 Finalize failed on flush for target "production_sqs": failed to send messages to sqs: operation error SQS: SendMessageBatch, https response error StatusCode: 400, RequestID: ..., BatchRequestTooLong: ...

Cause: the messages in one batch add up to more than the queue accepts. The target applies no size limit of its own, so the queue's MaximumMessageSize attribute decides, and the SQS API documents a ceiling of 1 MiB (1,048,576 bytes) both for a single message and for a whole batch. BatchRequestTooLong is the whole-batch form. A single oversized record fails as one entry instead, and shows up as failed to send N/10 messages to sqs.

Fix: lower max_events so fewer records travel together, trim the records in a pipeline, or raise the queue's MaximumMessageSize.

Retried until fixed, and refused identically each time, so treat this error as urgent: that worker stays on the same batch until the records get smaller or the queue attribute is raised.

"RequestThrottled"

Sender worker 6 execute() failed for production_sqs: target broken: failed to finalize target cache: failed to send messages to sqs: operation error SQS: SendMessageBatch, exceeded maximum number of attempts, 3, https response error StatusCode: 403, RequestID: ..., RequestThrottled: ...

Cause: the request rate on the queue is above its quota. exceeded maximum number of attempts, 3 means the AWS SDK already retried inside the call and was still throttled. Many workers sending small batches produce far more requests than fewer workers sending full ones.

Fix: keep max_events at 10, reduce the number of workers or spread delivery across several queues, then ask AWS for a quota increase if the rate is genuinely needed.

Retried until fixed, so delivery catches up once the rate drops. A 500 or 503 with a code such as ServiceUnavailable is an AWS-side problem and behaves the same way. Check AWS Health for your region and wait.

"request send failed", "no such host", or certificate errors

Failed to reinitialize target "production_sqs" (attempt 7). Reason: operation error STS: GetCallerIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts.us-east-1.amazonaws.com/": ...

Cause: the Director host cannot reach AWS. Two hostnames are involved: sts.us-east-1.amazonaws.com at startup, and sqs.us-east-1.amazonaws.com on every flush. The same error against the SQS hostname after a clean startup means only the second one is blocked. The ending is typically i/o timeout, no such host, or connection refused. With a TLS-intercepting proxy or a private CA it is typically x509: certificate signed by unknown authority.

Fix: allow outbound HTTPS on port 443 to both hostnames for your region. Set HTTPS_PROXY and NO_PROXY in the Director service environment when a proxy is required. This target has no TLS settings, so a proxy CA has to be installed in the host trust store, or the AWS endpoints excluded from the proxy. Raise timeout above the default of 30 only if the link is genuinely slow.

Retried until fixed.

note

endpoint replaces the address used for the credential check as well as the one used for SQS. LocalStack and ElasticMQ work because they answer both, but an endpoint that serves SQS alone fails at startup with an STS error even though the queue itself is reachable. An http:// endpoint against real AWS is refused with InvalidSecurity.

FIFO queues are not supported

Symptom: with a queue whose name ends in .fifo nothing is delivered, and every flush fails either as failed to send N/10 messages to sqs or as an error on the whole SendMessageBatch call.

Cause: a FIFO queue requires a message group id on every message. This target does not set one, so SQS refuses the messages.

Fix: send to a standard queue. Create a queue whose name does not end in .fifo, then point queue or queue_url at it.

Retried until fixed, and nothing ever reaches a FIFO queue.

The target is connected but nothing arrives in the queue

Check these in order.

1. debug.dont_send_logs is enabled. Events are processed by the pipeline and then discarded silently before anything is buffered for sending. No error is logged and no counter moves, so the target looks healthy and idle. With debug.status also enabled, one line is written at startup: Log sending is disabled for this target (production_sqs). Logs will be processed by the pipeline but will not be sent to the target. With debug.status off there is no trace at all. Remove dont_send_logs, which is a testing switch.

2. Messages go out under a different identity. If only one of key and secret has a value, both are ignored and the host's own credentials are used. Delivery may still work, but under the instance or task role of the host rather than the credentials you entered. See the entry on no EC2 IMDS role found above.

3. The same messages arrive repeatedly. After a partial failure the whole batch goes out again on every retry, so entries SQS already accepted are delivered more than once and the target's outgoing event count keeps climbing. Fix the failing entries as described above and the duplicates stop.