Skip to main content

Amazon CloudWatch Logs

Amazon AWS Logging

Synopsis

Creates a target that writes log messages to Amazon CloudWatch Logs with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits.

Schema

- name: <string>
description: <string>
type: amazoncloudwatch
pipelines: <pipeline[]>
status: <boolean>
properties:
key: <string>
secret: <string>
session: <string>
region: <string>
endpoint: <string>
log_group: <string>
log_stream: <string>
max_events: <numeric>
timeout: <numeric>
field_format: <string>
debug:
status: <boolean>
dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be amazoncloudwatch
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

AWS Credentials

FieldRequiredDefaultDescription
keyN*-AWS access key ID for authentication
secretN*-AWS secret access key for authentication
sessionN-Optional session token for temporary credentials
regionY-AWS region (e.g., us-east-1, eu-west-1)
endpointN-Custom endpoint URL, used for both CloudWatch Logs and the credential check (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Log Configuration

FieldRequiredDefaultDescription
log_groupY-CloudWatch Logs log group name
log_streamY-CloudWatch Logs log stream name
max_eventsN10000Maximum number of events per batch (1-10000)
timeoutN30Connection timeout in seconds
field_formatN-Data normalization format. See applicable Normalization section
note

Amazon CloudWatch Logs supports a maximum of 10,000 log events per PutLogEvents request. The max_events parameter must be between 1 and 10,000.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging
debug.dont_send_logsNfalseProcess logs but don't send to target (testing)

Details

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM ActionPurpose
sts:GetCallerIdentityValidate credentials at initialization
logs:CreateLogGroupCreate the log group. Called at every initialization, including when the group already exists
logs:CreateLogStreamCreate the log stream. Called at every initialization, including when the stream already exists
logs:PutLogEventsSend log events to the stream

Minimum IAM policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "STSIdentity",
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
},
{
"Sid": "CloudWatchLogsWrite",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:REGION:ACCOUNT_ID:log-group:LOG_GROUP_NAME",
"arn:aws:logs:REGION:ACCOUNT_ID:log-group:LOG_GROUP_NAME:log-stream:*"
]
}
]
}

Log Groups and Streams

CloudWatch Logs organizes log data into log groups and log streams:

Log Groups

  • Container for log streams
  • Define retention, monitoring, and access control settings
  • Created by the target at every initialization. An existing group is left as it is

Log Streams

  • Sequence of log events from the same source
  • Events must be in chronological order within a stream
  • Created by the target at every initialization. An existing stream is left as it is

Batch Processing

The target accumulates messages in memory and sends them in batches using the PutLogEvents API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10,000 log events per request.

Sequence Tokens

CloudWatch Logs no longer requires a sequence token on PutLogEvents, so no ordering state is carried between batches. Events within a single batch should still be in chronological order. See Troubleshooting for the error CloudWatch Logs returns when they are not.

Timestamps

Log events are sent with timestamps in milliseconds. The target automatically converts each event's epoch timestamp (seconds plus nanosecond precision) to milliseconds as required by CloudWatch Logs.

Data Retention

CloudWatch Logs retains log data indefinitely by default. You can configure retention periods from 1 day to 10 years at the log group level through the AWS Console or API.

Encryption

CloudWatch Logs encrypts log data at rest by default. All connections to CloudWatch Logs use HTTPS endpoints for encryption in transit.

Error Handling

The target handles common CloudWatch Logs errors:

  • Creates the log group and the log stream at every initialization, whether or not they already exist
  • Handles ResourceAlreadyExistsException when the group or the stream is already there
  • Surfaces events that CloudWatch silently rejects. CloudWatch accepts a PutLogEvents request (HTTP 200) while discarding events that fall outside its accepted time window (too old, too new, or expired); the target reports these rejected batches as warnings so they are not mistaken for successful delivery

Examples

Basic Configuration

The minimum configuration for a CloudWatch Logs target:

targets:
- name: basic_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "app-server-01"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
- name: iam_cloudwatch
type: amazoncloudwatch
properties:
region: "us-east-1"
log_group: "application-logs"
log_stream: "app-server-01"
note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate CloudWatch Logs permissions attached.

High Volume Logs

Configuration optimized for high-volume data:

targets:
- name: high_volume_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "high-volume-logs"
log_stream: "data-pipeline"
max_events: 10000
timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
- name: temp_creds_cloudwatch
type: amazoncloudwatch
properties:
key: "ASIATEMP1234567890AB"
secret: "tempSecretKeyExample1234567890"
session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
region: "us-west-2"
log_group: "temporary-logs"
log_stream: "session-logs"

With Field Normalization

Using field normalization for standard format:

targets:
- name: normalized_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "normalized-logs"
log_stream: "structured-events"
field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
- name: reliable_cloudwatch
type: amazoncloudwatch
pipelines:
- checkpoint
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "critical-logs"
log_stream: "production-events"
max_events: 1000

Multiple Applications

Configuration for different application log streams:

targets:
- name: web_server_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "web-server"

- name: api_server_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "api-server"

Multiple Regions

Configuration for CloudWatch Logs in different region:

targets:
- name: eu_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "eu-west-1"
log_group: "eu-application-logs"
log_stream: "eu-server-01"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
- name: scheduled_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "scheduled-logs"
log_stream: "batch-events"
max_events: 5000
interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
- name: debug_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "test-logs"
log_stream: "debug-stream"
debug:
status: true
dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing:

targets:
- name: local_cloudwatch
type: amazoncloudwatch
properties:
key: "test"
secret: "test"
region: "us-east-1"
endpoint: "http://localhost:4566"
log_group: "local-test-logs"
log_stream: "local-stream"

Production Configuration

Configuration for production with optimal settings:

targets:
- name: production_cloudwatch
type: amazoncloudwatch
pipelines:
- checkpoint
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "/aws/application/production"
log_stream: "app-cluster-01"
max_events: 10000
timeout: 60
field_format: "cim"

Container Logs

Configuration for containerized application logs:

targets:
- name: container_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "/ecs/application"
log_stream: "container-instance-01"
max_events: 8000

Troubleshooting

This section covers the errors you are most likely to see with the amazoncloudwatch target, what causes each one, and how to fix it. See Target Delivery Errors for how Director logs and retries target failures.

Where to look:

  • Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
  • The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

One fact shapes most of the entries below. Before it sends anything, the target checks its credentials with sts:GetCallerIdentity and then calls CreateLogGroup and CreateLogStream. It does this at startup and again after every failed delivery, so credential and permission problems appear at startup rather than on the first batch.

Which permission is missing?

Match the error you see against this table first. The identity is the one behind key and secret, or the role Director picks up from the host when either field is empty.

Error textMissing IAM actionAllowed on
failed to create log group: ... AccessDeniedExceptionlogs:CreateLogGroupThe log group, arn:aws:logs:us-east-1:000000000000:log-group:application-logs
failed to create log stream: ... AccessDeniedExceptionlogs:CreateLogStreamThe streams in that group, arn:aws:logs:us-east-1:000000000000:log-group:application-logs:log-stream:*
failed to put log events to cloudwatch: ... AccessDeniedExceptionlogs:PutLogEventsThe same stream resource as logs:CreateLogStream

Minimum policy, with the region, the account ID and the log group name replaced by your own:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:000000000000:log-group:application-logs",
"arn:aws:logs:us-east-1:000000000000:log-group:application-logs:log-stream:*"
]
}
]
}
warning

logs:CreateLogGroup and logs:CreateLogStream are required even when you pre-create the group and the stream. The target issues both calls on every initialization and only ignores the answer that says the resource already exists. A policy that grants logs:PutLogEvents alone never gets the target started.

"failed to create log group" or "failed to create log stream"

[Error] [director] [target-<target id>] [basic_cloudwatch] Failed to reinitialize target "basic_cloudwatch" (attempt 2). Reason: failed to create log group: operation error CloudWatch Logs: CreateLogGroup, https response error StatusCode: ..., RequestID: ..., AccessDeniedException: ...

Cause: the code at the end of the line says which problem it is. AccessDeniedException is the missing permission from the table above, and you see it even when the group already exists. InvalidParameterException means the name is not accepted: a log group name may be up to 512 characters of a-z, A-Z, 0-9, _, -, /, . and # and may not start with aws/, and a log stream name may be up to 512 characters and may not contain : or *. LimitExceededException means the account has reached its log group quota. OperationAbortedException means another Director created the same group at the same moment, and it clears by itself. ThrottlingException on CreateLogStream appears when many Director instances start at once, because that call is limited to 50 requests per second.

Fix: grant the action on the resource shown in the table, or correct log_group and log_stream. For a quota, raise it in Service Quotas. Nothing is sent while the target cannot start. Incoming data waits in the Director queue and is retried until you fix the cause.

"operation error STS: GetCallerIdentity" with an invalid token or signature

Failed to reinitialize target "basic_cloudwatch" (attempt 5). Reason: operation error STS: GetCallerIdentity, https response error StatusCode: ..., RequestID: ..., api error InvalidClientTokenId: ...

Cause: AWS refused the credentials before any permission was checked. The code names the field. Typically InvalidClientTokenId means key is wrong, deleted or deactivated, SignatureDoesNotMatch means secret is wrong, and ExpiredToken means session has expired.

Fix: correct key, secret and session, and rotate the access key if it was deactivated. session is only used when key and secret are both filled in. Credentials withdrawn after the target started show up on a batch first, typically as UnrecognizedClientException, and the next initialization then reports the code above. Retried until fixed, and nothing is lost in the meantime.

"no EC2 IMDS role found" or other credential chain errors

Failed to reinitialize target "basic_cloudwatch" (attempt 9). Reason: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, ...

Cause: no static credentials were used, so Director fell back to the standard AWS credential chain, and the chain found no usable role. The most common surprise is a half-configured pair: when either key or secret is empty, the other one is ignored without a message and the chain is used instead. Otherwise the Director host has no instance or task role, or the instance metadata service is unreachable. Containers on EC2 need an IMDSv2 hop limit of at least 2 to reach it.

Fix: set both key and secret, or set neither and attach a role to the host with the policy above. The text after failed to refresh cached credentials, names the provider the chain tried last, which tells you whether it looked for an instance role, a task role or a web identity. Retried until fixed.

warning

A half-configured pair does not always fail. If the host does have a role, the target authenticates as that role instead, and your events go to whatever account and log group that role can reach, with no error anywhere. Check the identity when events do not arrive where you expect them.

"failed to resolve access key" and other credential reference errors

Failed to reinitialize target "basic_cloudwatch" (attempt 7). Reason: failed to resolve access key: credential: env variable "AWS_ACCESS_KEY_ID" is not set

Cause: key, secret or session holds a ${ENV} or $secret{...} reference that could not be resolved. The first part names the field: failed to resolve access key, failed to resolve secret key or failed to resolve session token. The part after credential: says why:

TextMeaning
env variable "..." is not setThe variable is not exported to the Director service user. A variable set in your own shell is not visible to the service
failed to parse token "...": not a valid secret tokenThe $secret{...} syntax is malformed
store "..." not found in configurationThe store name in $secret{...} does not match a configured credential store
provider type "..." is not registered, or no credentials configuredThe store's provider type is misspelled, or no credential store is defined at all

Fix: export the variable for the service, or correct the reference or the store definition. Nothing is sent while this persists, and it is retried until fixed.

"Invalid Configuration: Missing Region"

Failed to reinitialize target "basic_cloudwatch" (attempt 1). Reason: operation error STS: GetCallerIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region

Cause: region is empty, and the Director host supplies no region either through AWS_REGION or a shared AWS profile. The field is not checked when you save the target, so an empty value only surfaces here.

Fix: set region to the region of the log group, for example us-east-1. A wrong but valid region is harder to spot than an empty one: the target creates the group and the stream in that region and delivers successfully, so check the region selector in the CloudWatch console before concluding that nothing arrived. Retried until fixed.

"failed to put log events to cloudwatch" with AccessDeniedException or ResourceNotFoundException

Sender worker 3 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, https response error StatusCode: ..., RequestID: ..., AccessDeniedException: ...

Cause: AccessDeniedException on a target that started normally means the policy covers the log group but not the streams inside it. logs:PutLogEvents must be allowed on the ...:log-stream:* resource, not only on the group. ResourceNotFoundException means the group or the stream was deleted after the target started. That one usually clears by itself, because the next initialization recreates both, so if it keeps coming back look for retention or lifecycle automation that removes the stream.

Fix: extend the policy as shown above. Retried until fixed and no data is lost, but every failed batch also reinitializes the target, so the connection status flips between connected and failed while this lasts.

"failed to put log events to cloudwatch" with InvalidParameterException

Sender worker 3 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, https response error StatusCode: ..., RequestID: ..., InvalidParameterException: ...

Cause: CloudWatch Logs rejected the batch itself. The target sends events in the order they arrive and does not reorder them, so the usual reasons are events that are not in chronological order within the batch, or a batch whose oldest and newest event are more than 24 hours apart. Both happen when a source replays old data or stamps records with mixed time zones. If the log group is encrypted with a customer-managed key, a key that is disabled or no longer usable typically produces the same code.

Fix: add a pipeline that sets @timestamp consistently, or that filters records whose timestamps are far from the others. Lowering max_events, or setting interval so that a batch covers a short period, keeps the batch inside the 24-hour span. For an encrypted group, check the key.

warning

This batch is never dropped. The same events are rebuilt and sent again every few seconds, with a full reinitialization each time, so the payload never clears and everything queued behind it on that worker waits. Correct the timestamps or the key, and the queue drains on the next attempt.

"failed to put log events to cloudwatch" with ThrottlingException

Sender worker 4 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, exceeded maximum number of attempts, 3, ...

Cause: the account's PutLogEvents quota per second is exhausted. The exceeded maximum number of attempts, 3 part means the AWS SDK already retried the call three times with backoff before Director logged anything.

Fix: request a higher quota in Service Quotas, reduce the number of targets and workers writing into the same account, or set interval so the target sends fewer, larger batches. Each failed batch also repeats the identity check and the two create calls, which adds to the load, so raising interval is the quickest relief. Retried until fixed.

"no such host", "i/o timeout", or certificate errors

Sender worker 2 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, dial tcp: lookup logs.us-east-1.amazonaws.com: no such host

Cause: Director could not complete the HTTPS request. The text at the end of the line is the transport error. You typically see no such host for a DNS failure, i/o timeout or connection refused for blocked egress, proxyconnect tcp when the proxy itself is unreachable, and x509: certificate signed by unknown authority when a TLS-intercepting proxy is in the path. The same causes at startup read as Failed to reinitialize target ... Reason: operation error STS: GetCallerIdentity, ....

Fix: allow outbound HTTPS (443) from the Director host to sts.us-east-1.amazonaws.com and to logs.us-east-1.amazonaws.com, substituting your region. Both are needed, because the identity check runs at every initialization. If you use a proxy, set HTTPS_PROXY for the Director service. Loopback and link-local addresses always bypass the proxy, so instance role credentials keep working behind a mandatory proxy. The target has no TLS settings of its own and trusts the operating system certificate store, so install an intercepting proxy's CA certificate at the OS level. Note that endpoint replaces the address for the credential check as well, so a local emulator that does not answer STS calls never gets past startup. Retried until fixed.

"record rejected by target: event size ... exceeds the CloudWatch per-event limit"

Sender worker 1 execute() failed for basic_cloudwatch: record rejected by target: event size 300026 bytes exceeds the CloudWatch per-event limit of 262144 bytes
Sender worker 1 deterministic failure for basic_cloudwatch after 4 attempts — dropping (giving up): record rejected by target: event size 300026 bytes exceeds the CloudWatch per-event limit of 262144 bytes

Cause: one record is too large. The limit is 262,144 bytes per event, and every event carries 26 bytes of overhead on top of its message.

Fix: trim or split the record in a pipeline before it reaches the target, for example by dropping the oversized field. That one record is dropped after 4 deliveries, while the records ahead of it in the same payload are sent and the rest of the traffic keeps flowing. A batch also has a ceiling of 1,048,576 bytes in total: when the next event does not fit, the target sends what it has and starts a new batch, which is normal and is not logged.

"CloudWatch rejected log events in ..." with events missing from the stream

[Warning] [director] [target-<target id>] [basic_cloudwatch] CloudWatch rejected log events in application-logs/app-server-01 (batch size 5000): tooOldEndIndex=312 tooNewStartIndex=-1 expiredEndIndex=-1

Cause: CloudWatch Logs accepted the request and then discarded individual events because of their timestamps. The three numbers say which events were refused: tooOldEndIndex marks events older than 14 days, expiredEndIndex marks events older than the log group's retention setting, and tooNewStartIndex marks events more than 2 hours in the future. A -1 means nothing was refused for that reason. A record whose timestamp could not be parsed becomes 1970-01-01, so a parsing problem in a pipeline also lands here as too old.

How to notice it: the delivery counts as successful, so the connection status stays connected and no error is logged. The warning above is the only trace. Search the Director logs for CloudWatch rejected log events, and watch the target's dropped counter in the stats view. A dropped counter that climbs while the target looks healthy is this.

Fix: correct the timestamps at the source, or add a pipeline that sets @timestamp to the ingest time for feeds that replay old data. Extend the log group's retention if the events are genuinely older than it, and check the clock and time zone of the sending host for events in the future. The events that were refused are dropped silently. They are counted as dropped and are never retried.

The target is healthy but nothing arrives in CloudWatch

Check these in order. Remember first that with interval or cron set, events accumulate until the next tick, so a quiet stream may simply not have been flushed yet.

  1. debug.dont_send_logs is enabled. Events are processed and then discarded instead of being sent. The notice Log sending is disabled for this target is only written when debug.status is also enabled, so with debug.status: false this is completely silent. The Debug Configuration example on this page sets both. Remove the flag.

  2. The events are being refused for their timestamps. See the entry above, and check the dropped counter.

  3. The data went somewhere else. If only one of key and secret is set, the target authenticates as the host's role, which may belong to another account. A valid but wrong region puts the group and the stream in that region. Both deliver successfully.

"ValidateConfig failed" with a configuration reason

ValidateConfig failed for target "basic_cloudwatch": log_group is required for amazon cloudwatch target

Cause: the target was rejected when the configuration was read. The reason is log_group is required for amazon cloudwatch target, log_stream is required for amazon cloudwatch target, or max_events must be between 1 and 10000, got 20000 with your own value.

Fix: set the missing field, or bring max_events into the range 1 to 10000. Nothing is sent while the configuration is rejected, and Director re-reads it about every 30 seconds, so the target starts as soon as the value is valid. Note that max_events: 0, and any value that is not a number, is accepted and silently becomes the default of 10000.