Amazon SNS

Amazon AWS Message Queue

Synopsis

Creates a target that writes log messages to Amazon Simple Notification Service (SNS) with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits. Amazon SNS is a fully managed pub/sub messaging service that enables message delivery to multiple subscribers including SQS queues, Lambda functions, HTTP endpoints, email, and mobile push notifications.

Schema

- name: <string>
  description: <string>
  type: amazonsns
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    topic: <string>
    subject: <string>
    max_events: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazonsns
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom SNS endpoint URL (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Topic Configuration

Field	Required	Default	Description
topic	Y	-	SNS topic ARN (e.g., arn:aws:sns:us-east-1:123456789012:my-topic)
subject	N	"DataStream Log"	Default subject line for messages
max_events	N	10	Maximum number of events per batch (1-10)
timeout	N	30	Timeout in seconds for each publish request
field_format	N	-	Data normalization format. See applicable Normalization section

note

Amazon SNS supports a maximum of 10 messages per PublishBatch request. The max_events parameter must be between 1 and 10.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Amazon SNS is a fully managed pub/sub messaging service that enables message fanout to multiple subscribers. This target allows you to publish log messages to SNS topics for distribution to various endpoints.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM Action	Purpose
sts:GetCallerIdentity	Validate credentials at initialization
sns:Publish	Publish messages to the topic

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "STSIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "SNSPublish",
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC_NAME"
    }
  ]
}

note

sns:Publish is the action that covers both the single and the batch publish calls. There is no separate sns:PublishBatch action, so a policy naming that string grants nothing.

Topic Types

Amazon SNS supports two types of topics:

Standard Topics

• Best-effort message ordering
• At-least-once delivery
• Nearly unlimited throughput
• Default and most common type

FIFO Topics

• Strict message ordering
• Exactly-once message delivery
• Limited to 300 messages per second (3000 with batching)
• Topic name must end with .fifo
• Only compatible with FIFO SQS queues
• Require a message group ID on every message

warning

Use a standard topic. This target does not send a message group ID, which a FIFO topic requires on every message, so a topic whose name ends in .fifo rejects every batch. See Publishing to a FIFO topic under Troubleshooting.

Subscription Types

SNS topics can deliver messages to multiple subscription types:

• Amazon SQS queues
• AWS Lambda functions
• HTTP/HTTPS endpoints
• Email addresses
• SMS text messages
• Mobile push notifications (iOS, Android, etc.)

Message Properties

SNS messages have the following characteristics:

• Maximum message size: 256 KB
• Message retention: Messages are not stored, delivered immediately
• Message attributes: Up to 10 message attributes per message
• Message filtering: Subscribers can filter messages based on attributes

Batch Processing

The target accumulates messages in memory and sends them in batches using the PublishBatch API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10 messages per request (Amazon SNS limit).

Message Delivery

SNS attempts to deliver messages to all subscribed endpoints. Delivery behavior depends on the subscription type:

• SQS: Guaranteed delivery with retry
• Lambda: Synchronous or asynchronous invocation
• HTTP/HTTPS: Configurable retry policies
• Email/SMS: Best-effort delivery

Encryption

SNS supports server-side encryption using AWS KMS. Messages are encrypted in transit and at rest. All connections to SNS use HTTPS endpoints.

Message Filtering

Subscribers can use filter policies to receive only a subset of messages published to a topic. This reduces unnecessary message processing and costs.

Integration with AWS Services

SNS integrates with other AWS services:

• Amazon SQS for reliable queuing
• AWS Lambda for serverless processing
• Amazon CloudWatch for monitoring and alarms
• AWS Step Functions for workflow orchestration
• Amazon EventBridge for event routing

Examples

Basic Configuration

The minimum configuration for an SNS target:

targets:
  - name: basic_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:application-logs"

With Custom Subject

Configuration with custom message subject:

targets:
  - name: custom_subject_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:application-logs"
      subject: "Application Error Alert"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
  - name: iam_sns
    type: amazonsns
    properties:
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:application-logs"

note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate SNS permissions attached.

High Throughput

Configuration optimized for high-volume data:

targets:
  - name: high_volume_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:high-volume-logs"
      max_events: 10
      timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
  - name: temp_creds_sns
    type: amazonsns
    properties:
      key: "ASIATEMP1234567890AB"
      secret: "tempSecretKeyExample1234567890"
      session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
      region: "us-west-2"
      topic: "arn:aws:sns:us-west-2:123456789012:temporary-logs"

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:normalized-logs"
      field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
  - name: reliable_sns
    type: amazonsns
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:critical-logs"
      max_events: 5

Multiple Regions

Configuration for SNS topic in different region:

targets:
  - name: eu_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      topic: "arn:aws:sns:eu-west-1:123456789012:eu-application-logs"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:scheduled-logs"
      max_events: 10
      interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:test-logs"
      debug:
        status: true
        dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing (e.g., LocalStack):

targets:
  - name: local_sns
    type: amazonsns
    properties:
      key: "test"
      secret: "test"
      region: "us-east-1"
      endpoint: "http://localhost:4566"
      topic: "arn:aws:sns:us-east-1:000000000000:local-test-topic"

Fanout to Multiple Services

Configuration for fanout pattern (SNS topic with multiple SQS queue subscriptions):

targets:
  - name: fanout_sns
    type: amazonsns
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:fanout-logs"
      subject: "Log Distribution"

note

Configure SNS topic subscriptions (SQS, Lambda, HTTP, Email, etc.) separately in the AWS Console or via infrastructure as code.

Production Configuration

Configuration for production with optimal settings:

targets:
  - name: production_sns
    type: amazonsns
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      topic: "arn:aws:sns:us-east-1:123456789012:production-logs"
      subject: "Production Log Event"
      max_events: 10
      timeout: 60
      field_format: "cim"

Troubleshooting

This section covers the errors you are most likely to see with the amazonsns target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission is missing?

Match the error you see against this table. The identity checked is the one behind key and secret, or the instance role when those are not set.

Error text	Missing IAM action	Allowed on
api error AuthorizationError on SNS: PublishBatch	sns:Publish	The topic ARN you set in topic
api error KMSAccessDenied, KMSDisabled or KMSNotFound on SNS: PublishBatch	kms:GenerateDataKey and kms:Decrypt, typically	The KMS key an encrypted topic uses
api error InvalidClientTokenId or SignatureDoesNotMatch on STS: GetCallerIdentity	None. The credentials are rejected before any permission is checked	Not applicable

Minimum policy for delivery:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "STSIdentity", "Effect": "Allow", "Action": "sts:GetCallerIdentity", "Resource": "*" },
    { "Sid": "SNSPublish", "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC_NAME" }
  ]
}

sns:Publish covers both the single and the batch publish calls, so it is the only action granted above. There is no separate sns:PublishBatch action, and a policy naming that string grants nothing.

For a topic in another account, an identity-based policy is not enough on its own. The topic's own access policy must also allow the publishing account.

"failed to resolve access key" or "failed to resolve secret key"

[Error] [director] [target-<target id>] [production_sns] Failed to reinitialize target "production_sns" (attempt 2). Reason: failed to resolve secret key: credential: env variable "AWS_SECRET_ACCESS_KEY" is not set

Cause: key, secret or session holds a ${ENV_VAR} or $secret{...} reference that could not be resolved. The variable is not set for the Director service, the token is malformed, or the secret store it names is not configured. A store that does not answer within 30 seconds fails the same way, with context deadline exceeded.

Fix: export the variable for the Director service itself, not only in your shell, and restart the service so it inherits the value. For a $secret{...} reference, check that the store is defined and that the secret ID exists in it.

The target does not start, so nothing is sent, and incoming data waits in the queue until the reference resolves.

"InvalidClientTokenId", "SignatureDoesNotMatch" or "ExpiredTokenException"

Reason: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: ..., api error InvalidClientTokenId: ...

Cause: Director validates the credentials with a call to sts:GetCallerIdentity before it builds the SNS client, and AWS rejected them. The code tells you which field is wrong. InvalidClientTokenId means key is wrong or deleted. SignatureDoesNotMatch means secret does not match key, or the host clock is more than a few minutes out. ExpiredTokenException means the temporary session token has expired. RegionDisabledException means STS is not enabled in the opt-in region you set.

Fix: issue a new access key pair and set both key and secret. For temporary credentials, refresh session, or remove it and use permanent keys or an instance role. Check time synchronization on the Director host for a signature error.

The target does not start, so nothing is sent until the credentials are accepted.

"no EC2 IMDS role found", or messages are published by the wrong identity

Reason: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, ...

Cause: key and secret are used only when both resolve to a non-empty value. If either one is missing, both are ignored silently and the default AWS credential chain is used instead: environment variables, a shared profile, then the instance or task role. Off AWS, or where the instance metadata service is blocked, that chain ends in this error. On AWS it succeeds, and the target then publishes as the instance role instead of the identity you intended. A typo in one of the two property names looks exactly like this.

Fix: set both key and secret, or set neither and attach a role that allows sns:Publish to the instance or task. To see which identity is actually in use, read the principal ARN out of an AuthorizationError message, which typically names it.

Nothing is sent while the chain fails. If the chain succeeds under an unintended identity, the data is published under that identity instead.

"Invalid Configuration: Missing Region"

Reason: operation error STS: GetCallerIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region

Cause: region is empty, and no region is set in the environment or the shared profile of the Director service account. The message comes from the AWS SDK and never names the region property.

Fix: set region, for example us-east-1. Use the region that appears in the ARN you set in topic, because requests go to the endpoint for region.

Nothing is sent until a region resolves.

"AuthorizationError" when publishing

Sender worker 3 execute() failed for production_sns: failed to publish messages to sns: operation error SNS: PublishBatch, https response error StatusCode: 403, RequestID: ..., api error AuthorizationError: ...

Cause: the credentials are valid and the target started, but the identity may not publish to this topic. The message typically continues with is not authorized to perform and names both the principal and the topic ARN.

Fix: grant sns:Publish on that topic ARN to that principal, as in the policy above. For a topic in another account, add the principal to the topic's access policy as well.

The batch is retried until the permission is in place, so nothing is lost in the meantime.

"NotFound" when publishing

Sender worker 1 execute() failed for production_sns: failed to publish messages to sns: operation error SNS: PublishBatch, https response error StatusCode: 404, RequestID: ..., api error NotFound: ...

Cause: the region the request went to has no such topic. The message typically reads Topic does not exist. Either the ARN in topic has a typo or names a deleted topic, or its region segment is not the value in region. A topic in eu-west-1 published with region: "us-east-1" fails this way, because the request goes to the us-east-1 endpoint.

Fix: copy the ARN from the SNS console into topic, and set region to the region inside that ARN.

Retried until the ARN and the region agree, so nothing is lost.

"InvalidParameter" or "ParameterValueInvalid"

Sender worker 1 execute() failed for production_sns: failed to publish messages to sns: operation error SNS: PublishBatch, https response error StatusCode: 400, RequestID: ..., api error InvalidParameter: ...

Cause: SNS rejected something in the request itself, and the text after the code typically names the parameter. Common causes are a topic value that is not a well-formed topic ARN, a FIFO topic (see the next entry), a subject that SNS will not accept, or a record whose body is empty or not valid UTF-8. SNS typically limits subject to 100 ASCII characters with no leading whitespace, and the target sends the value on every message without checking it first.

Fix: correct the property named in the message. For record bodies, drop or repair the offending records in a pipeline before the target.

A batch rejected this way is retried, not dropped. The same records rebuild the same batch on every attempt, so a rejection that repeats blocks the worker until you change the configuration or the data. Check this entry first after changing subject or topic.

Publishing to a FIFO topic

Symptom: every batch sent to a topic whose name ends in .fifo is rejected with the InvalidParameter error above, and the message typically names MessageGroupId. Nothing is ever delivered to the topic.

Fix: publish to a standard topic. A FIFO topic requires a message group ID on every message, and this target does not send one. Set topic to the ARN of a standard topic and subscribe the same endpoints to it.

Nothing reaches the topic, and the rejected batches are retried until topic points at a standard topic.

"BatchRequestTooLong", or "failed to publish 1/10 messages to sns"

Sender worker 4 execute() failed for production_sns: failed to publish messages to sns: operation error SNS: PublishBatch, https response error StatusCode: 400, RequestID: ..., api error BatchRequestTooLong: ...

Sender worker 2 Finalize failed on flush for target "production_sns": failed to publish 1/10 messages to sns

Cause: SNS accepts at most 256 KB in one message, and at most 256 KB for all the messages of one batch added together. The target does not measure the records it buffers, so ten records of 30 KB each already exceed the batch limit. BatchRequestTooLong rejects the whole call. The second form means SNS took the call but refused some of the messages in it, usually one oversized or malformed record; the per-message reason is not part of that log line.

Fix: lower max_events so fewer records go into each batch, and trim large fields in a pipeline so single records stay well under 256 KB. Set max_events to a number: a value that is not numeric is accepted and treated as the default of 10. To read the per-message error code behind the second form, turn on delivery status logging for the topic, or use CloudTrail.

Both forms are retried, and the same records rebuild the same batch every time, so act on the size rather than waiting. On a partial failure the messages that did succeed are published again on every retry, so make the subscribers tolerant of duplicates.

"Throttled"

Sender worker 2 execute() failed for production_sns: failed to publish messages to sns: operation error SNS: PublishBatch, https response error StatusCode: 429, RequestID: ..., api error Throttled: ...

Cause: the publish rate for the account or the topic was exceeded. The message typically reads Rate exceeded.

Fix: request a higher publish quota for SNS, or set interval so batches are flushed on a schedule instead of as soon as they fill. See Scheduling and Pool Behavior.

Throttled batches are retried until they are accepted, so nothing is lost, but the queue grows while the throttling lasts.

"no such host", "i/o timeout" or "certificate signed by unknown authority"

Reason: operation error STS: GetCallerIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts.us-east-1.amazonaws.com/": dial tcp: lookup sts.us-east-1.amazonaws.com: no such host

Cause: the Director host cannot reach AWS. The exceeded maximum number of attempts, 3 part means the AWS SDK already retried the call before the error surfaced. The same shapes appear for SNS: PublishBatch once the target is running.

Fix: allow outbound HTTPS on port 443 from the Director host to sts.<region>.amazonaws.com and sns.<region>.amazonaws.com. If you use a proxy, set HTTP_PROXY, HTTPS_PROXY and NO_PROXY for the Director service; a proxyconnect tcp error means the proxy itself could not be reached. For x509: certificate signed by unknown authority, install the certificate authority of your intercepting proxy into the host trust store, which is the only trust source this target uses. On a slow link, raise timeout, which bounds each publish call.

Retried until the route is restored, so nothing is lost.

The target is healthy but nothing arrives at the subscribers

Check these in order.

1. debug.dont_send_logs is enabled. Records are processed by the pipelines and then discarded instead of published. There is no error, and no counter moves. When debug.status is also enabled, a hint is logged once at startup:

   [Debug] [director] [target-<target id>] [production_sns] Log sending is disabled for this target (production_sns). Logs will be processed by the pipeline but will not be sent to the target.

   Set debug.dont_send_logs: false.

2. SNS accepted the messages but did not fan them out. Once SNS returns a message ID, Director counts the event as delivered and sees nothing that happens after that. Subscription filter policies, subscriptions still awaiting confirmation, an SQS queue policy that refuses the topic, and failing Lambda functions are all invisible here. Check the topic's NumberOfNotificationsFailed metric in CloudWatch, and turn on delivery status logging for the topic.

3. The messages go to a topic you are not watching. Confirm the account ID and the topic name inside topic, and that region matches that ARN.

4. Only one of key and secret is set. The target then publishes under the host's instance role, which may reach a different topic than you expect. See the entry on the instance role above.