Amazon CloudWatch Logs
Synopsis
Creates a target that writes log messages to Amazon CloudWatch Logs with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits.
Schema
- name: <string>
description: <string>
type: amazoncloudwatch
pipelines: <pipeline[]>
status: <boolean>
properties:
key: <string>
secret: <string>
session: <string>
region: <string>
endpoint: <string>
log_group: <string>
log_stream: <string>
max_events: <numeric>
timeout: <numeric>
field_format: <string>
debug:
status: <boolean>
dont_send_logs: <boolean>
Configuration
The following fields are used to define the target:
| Field | Required | Default | Description |
|---|---|---|---|
name | Y | Target name | |
description | N | - | Optional description |
type | Y | Must be amazoncloudwatch | |
pipelines | N | - | Optional post-processor pipelines |
status | N | true | Enable/disable the target |
AWS Credentials
| Field | Required | Default | Description |
|---|---|---|---|
key | N* | - | AWS access key ID for authentication |
secret | N* | - | AWS secret access key for authentication |
session | N | - | Optional session token for temporary credentials |
region | Y | - | AWS region (e.g., us-east-1, eu-west-1) |
endpoint | N | - | Custom endpoint URL, used for both CloudWatch Logs and the credential check (for testing or local development) |
* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.
Log Configuration
| Field | Required | Default | Description |
|---|---|---|---|
log_group | Y | - | CloudWatch Logs log group name |
log_stream | Y | - | CloudWatch Logs log stream name |
max_events | N | 10000 | Maximum number of events per batch (1-10000) |
timeout | N | 30 | Connection timeout in seconds |
field_format | N | - | Data normalization format. See applicable Normalization section |
Amazon CloudWatch Logs supports a maximum of 10,000 log events per PutLogEvents request. The max_events parameter must be between 1 and 10,000.
Scheduling
See Scheduling and Pool Behavior for interval and cron fields shared by all targets.
Debug Options
| Field | Required | Default | Description |
|---|---|---|---|
debug.status | N | false | Enable debug logging |
debug.dont_send_logs | N | false | Process logs but don't send to target (testing) |
Details
Authentication Methods
Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.
All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.
IAM Permissions
When using IAM role-based authentication, the following permissions are required:
| IAM Action | Purpose |
|---|---|
sts:GetCallerIdentity | Validate credentials at initialization |
logs:CreateLogGroup | Create the log group. Called at every initialization, including when the group already exists |
logs:CreateLogStream | Create the log stream. Called at every initialization, including when the stream already exists |
logs:PutLogEvents | Send log events to the stream |
Minimum IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "STSIdentity",
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
},
{
"Sid": "CloudWatchLogsWrite",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:REGION:ACCOUNT_ID:log-group:LOG_GROUP_NAME",
"arn:aws:logs:REGION:ACCOUNT_ID:log-group:LOG_GROUP_NAME:log-stream:*"
]
}
]
}
Log Groups and Streams
CloudWatch Logs organizes log data into log groups and log streams:
Log Groups
- Container for log streams
- Define retention, monitoring, and access control settings
- Created by the target at every initialization. An existing group is left as it is
Log Streams
- Sequence of log events from the same source
- Events must be in chronological order within a stream
- Created by the target at every initialization. An existing stream is left as it is
Batch Processing
The target accumulates messages in memory and sends them in batches using the PutLogEvents API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10,000 log events per request.
Sequence Tokens
CloudWatch Logs no longer requires a sequence token on PutLogEvents, so no ordering state is carried between batches. Events within a single batch should still be in chronological order. See Troubleshooting for the error CloudWatch Logs returns when they are not.
Timestamps
Log events are sent with timestamps in milliseconds. The target automatically converts each event's epoch timestamp (seconds plus nanosecond precision) to milliseconds as required by CloudWatch Logs.
Data Retention
CloudWatch Logs retains log data indefinitely by default. You can configure retention periods from 1 day to 10 years at the log group level through the AWS Console or API.
Encryption
CloudWatch Logs encrypts log data at rest by default. All connections to CloudWatch Logs use HTTPS endpoints for encryption in transit.
Error Handling
The target handles common CloudWatch Logs errors:
- Creates the log group and the log stream at every initialization, whether or not they already exist
- Handles
ResourceAlreadyExistsExceptionwhen the group or the stream is already there - Surfaces events that CloudWatch silently rejects. CloudWatch accepts a
PutLogEventsrequest (HTTP 200) while discarding events that fall outside its accepted time window (too old, too new, or expired); the target reports these rejected batches as warnings so they are not mistaken for successful delivery
Examples
Basic Configuration
The minimum configuration for a CloudWatch Logs target:
targets:
- name: basic_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "app-server-01"
With IAM Role
Configuration using IAM role authentication (no explicit credentials):
targets:
- name: iam_cloudwatch
type: amazoncloudwatch
properties:
region: "us-east-1"
log_group: "application-logs"
log_stream: "app-server-01"
When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate CloudWatch Logs permissions attached.
High Volume Logs
Configuration optimized for high-volume data:
targets:
- name: high_volume_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "high-volume-logs"
log_stream: "data-pipeline"
max_events: 10000
timeout: 60
With Temporary Credentials
Configuration using temporary session credentials:
targets:
- name: temp_creds_cloudwatch
type: amazoncloudwatch
properties:
key: "ASIATEMP1234567890AB"
secret: "tempSecretKeyExample1234567890"
session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
region: "us-west-2"
log_group: "temporary-logs"
log_stream: "session-logs"
With Field Normalization
Using field normalization for standard format:
targets:
- name: normalized_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "normalized-logs"
log_stream: "structured-events"
field_format: "cim"
With Checkpoint Pipeline
Configuration with checkpoint pipeline for reliability:
targets:
- name: reliable_cloudwatch
type: amazoncloudwatch
pipelines:
- checkpoint
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "critical-logs"
log_stream: "production-events"
max_events: 1000
Multiple Applications
Configuration for different application log streams:
targets:
- name: web_server_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "web-server"
- name: api_server_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "application-logs"
log_stream: "api-server"
Multiple Regions
Configuration for CloudWatch Logs in different region:
targets:
- name: eu_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "eu-west-1"
log_group: "eu-application-logs"
log_stream: "eu-server-01"
Scheduled Batching
Configuration with scheduled batch delivery:
targets:
- name: scheduled_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "scheduled-logs"
log_stream: "batch-events"
max_events: 5000
interval: "5m"
Debug Configuration
Configuration with debugging enabled:
targets:
- name: debug_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "test-logs"
log_stream: "debug-stream"
debug:
status: true
dont_send_logs: true
Local Development
Configuration with custom endpoint for local testing:
targets:
- name: local_cloudwatch
type: amazoncloudwatch
properties:
key: "test"
secret: "test"
region: "us-east-1"
endpoint: "http://localhost:4566"
log_group: "local-test-logs"
log_stream: "local-stream"
Production Configuration
Configuration for production with optimal settings:
targets:
- name: production_cloudwatch
type: amazoncloudwatch
pipelines:
- checkpoint
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "/aws/application/production"
log_stream: "app-cluster-01"
max_events: 10000
timeout: 60
field_format: "cim"
Container Logs
Configuration for containerized application logs:
targets:
- name: container_cloudwatch
type: amazoncloudwatch
properties:
key: "AKIAIOSFODNN7EXAMPLE"
secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region: "us-east-1"
log_group: "/ecs/application"
log_stream: "container-instance-01"
max_events: 8000
Troubleshooting
This section covers the errors you are most likely to see with the amazoncloudwatch target, what causes each one, and how to fix it. See Target Delivery Errors for how Director logs and retries target failures.
Where to look:
- Director logs. Target errors are tagged with the target name and carry
"Section":"SenderPool". The part afterReason:or after the last colon is the actual cause. - The target's connection status in the web interface. It shows the same reason as the log line, prefixed with
connection failed for <target name>:.
One fact shapes most of the entries below. Before it sends anything, the target checks its credentials with sts:GetCallerIdentity and then calls CreateLogGroup and CreateLogStream. It does this at startup and again after every failed delivery, so credential and permission problems appear at startup rather than on the first batch.
Which permission is missing?
Match the error you see against this table first. The identity is the one behind key and secret, or the role Director picks up from the host when either field is empty.
| Error text | Missing IAM action | Allowed on |
|---|---|---|
failed to create log group: ... AccessDeniedException | logs:CreateLogGroup | The log group, arn:aws:logs:us-east-1:000000000000:log-group:application-logs |
failed to create log stream: ... AccessDeniedException | logs:CreateLogStream | The streams in that group, arn:aws:logs:us-east-1:000000000000:log-group:application-logs:log-stream:* |
failed to put log events to cloudwatch: ... AccessDeniedException | logs:PutLogEvents | The same stream resource as logs:CreateLogStream |
Minimum policy, with the region, the account ID and the log group name replaced by your own:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:000000000000:log-group:application-logs",
"arn:aws:logs:us-east-1:000000000000:log-group:application-logs:log-stream:*"
]
}
]
}
logs:CreateLogGroup and logs:CreateLogStream are required even when you pre-create the group and the stream. The target issues both calls on every initialization and only ignores the answer that says the resource already exists. A policy that grants logs:PutLogEvents alone never gets the target started.
"failed to create log group" or "failed to create log stream"
[Error] [director] [target-<target id>] [basic_cloudwatch] Failed to reinitialize target "basic_cloudwatch" (attempt 2). Reason: failed to create log group: operation error CloudWatch Logs: CreateLogGroup, https response error StatusCode: ..., RequestID: ..., AccessDeniedException: ...
Cause: the code at the end of the line says which problem it is. AccessDeniedException is the missing permission from the table above, and you see it even when the group already exists. InvalidParameterException means the name is not accepted: a log group name may be up to 512 characters of a-z, A-Z, 0-9, _, -, /, . and # and may not start with aws/, and a log stream name may be up to 512 characters and may not contain : or *. LimitExceededException means the account has reached its log group quota. OperationAbortedException means another Director created the same group at the same moment, and it clears by itself. ThrottlingException on CreateLogStream appears when many Director instances start at once, because that call is limited to 50 requests per second.
Fix: grant the action on the resource shown in the table, or correct log_group and log_stream. For a quota, raise it in
"operation error STS: GetCallerIdentity" with an invalid token or signature
Failed to reinitialize target "basic_cloudwatch" (attempt 5). Reason: operation error STS: GetCallerIdentity, https response error StatusCode: ..., RequestID: ..., api error InvalidClientTokenId: ...
Cause: AWS refused the credentials before any permission was checked. The code names the field. Typically InvalidClientTokenId means key is wrong, deleted or deactivated, SignatureDoesNotMatch means secret is wrong, and ExpiredToken means session has expired.
Fix: correct key, secret and session, and rotate the access key if it was deactivated. session is only used when key and secret are both filled in. Credentials withdrawn after the target started show up on a batch first, typically as UnrecognizedClientException, and the next initialization then reports the code above. Retried until fixed, and nothing is lost in the meantime.
"no EC2 IMDS role found" or other credential chain errors
Failed to reinitialize target "basic_cloudwatch" (attempt 9). Reason: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, ...
Cause: no static credentials were used, so Director fell back to the standard AWS credential chain, and the chain found no usable role. The most common surprise is a half-configured pair: when either key or secret is empty, the other one is ignored without a message and the chain is used instead. Otherwise the Director host has no instance or task role, or the instance metadata service is unreachable. Containers on EC2 need an IMDSv2 hop limit of at least 2 to reach it.
Fix: set both key and secret, or set neither and attach a role to the host with the policy above. The text after failed to refresh cached credentials, names the provider the chain tried last, which tells you whether it looked for an instance role, a task role or a web identity. Retried until fixed.
A half-configured pair does not always fail. If the host does have a role, the target authenticates as that role instead, and your events go to whatever account and log group that role can reach, with no error anywhere. Check the identity when events do not arrive where you expect them.
"failed to resolve access key" and other credential reference errors
Failed to reinitialize target "basic_cloudwatch" (attempt 7). Reason: failed to resolve access key: credential: env variable "AWS_ACCESS_KEY_ID" is not set
Cause: key, secret or session holds a ${ENV} or $secret{...} reference that could not be resolved. The first part names the field: failed to resolve access key, failed to resolve secret key or failed to resolve session token. The part after credential: says why:
| Text | Meaning |
|---|---|
env variable "..." is not set | The variable is not exported to the Director service user. A variable set in your own shell is not visible to the service |
failed to parse token "...": not a valid secret token | The $secret{...} syntax is malformed |
store "..." not found in configuration | The store name in $secret{...} does not match a configured credential store |
provider type "..." is not registered, or no credentials configured | The store's provider type is misspelled, or no credential store is defined at all |
Fix: export the variable for the service, or correct the reference or the store definition. Nothing is sent while this persists, and it is retried until fixed.
"Invalid Configuration: Missing Region"
Failed to reinitialize target "basic_cloudwatch" (attempt 1). Reason: operation error STS: GetCallerIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region
Cause: region is empty, and the Director host supplies no region either through AWS_REGION or a shared AWS profile. The field is not checked when you save the target, so an empty value only surfaces here.
Fix: set region to the region of the log group, for example us-east-1. A wrong but valid region is harder to spot than an empty one: the target creates the group and the stream in that region and delivers successfully, so check the region selector in the CloudWatch console before concluding that nothing arrived. Retried until fixed.
"failed to put log events to cloudwatch" with AccessDeniedException or ResourceNotFoundException
Sender worker 3 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, https response error StatusCode: ..., RequestID: ..., AccessDeniedException: ...
Cause: AccessDeniedException on a target that started normally means the policy covers the log group but not the streams inside it. logs:PutLogEvents must be allowed on the ...:log-stream:* resource, not only on the group. ResourceNotFoundException means the group or the stream was deleted after the target started. That one usually clears by itself, because the next initialization recreates both, so if it keeps coming back look for retention or lifecycle automation that removes the stream.
Fix: extend the policy as shown above. Retried until fixed and no data is lost, but every failed batch also reinitializes the target, so the connection status flips between connected and failed while this lasts.
"failed to put log events to cloudwatch" with InvalidParameterException
Sender worker 3 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, https response error StatusCode: ..., RequestID: ..., InvalidParameterException: ...
Cause: CloudWatch Logs rejected the batch itself. The target sends events in the order they arrive and does not reorder them, so the usual reasons are events that are not in chronological order within the batch, or a batch whose oldest and newest event are more than 24 hours apart. Both happen when a source replays old data or stamps records with mixed time zones. If the log group is encrypted with a customer-managed key, a key that is disabled or no longer usable typically produces the same code.
Fix: add a pipeline that sets @timestamp consistently, or that filters records whose timestamps are far from the others. Lowering max_events, or setting interval so that a batch covers a short period, keeps the batch inside the 24-hour span. For an encrypted group, check the key.
This batch is never dropped. The same events are rebuilt and sent again every few seconds, with a full reinitialization each time, so the payload never clears and everything queued behind it on that worker waits. Correct the timestamps or the key, and the queue drains on the next attempt.
"failed to put log events to cloudwatch" with ThrottlingException
Sender worker 4 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, exceeded maximum number of attempts, 3, ...
Cause: the account's PutLogEvents quota per second is exhausted. The exceeded maximum number of attempts, 3 part means the AWS SDK already retried the call three times with backoff before Director logged anything.
Fix: request a higher quota in interval so the target sends fewer, larger batches. Each failed batch also repeats the identity check and the two create calls, which adds to the load, so raising interval is the quickest relief. Retried until fixed.
"no such host", "i/o timeout", or certificate errors
Sender worker 2 execute() failed for basic_cloudwatch: target broken: failed to finalize target cache: failed to put log events to cloudwatch: operation error CloudWatch Logs: PutLogEvents, dial tcp: lookup logs.us-east-1.amazonaws.com: no such host
Cause: Director could not complete the HTTPS request. The text at the end of the line is the transport error. You typically see no such host for a DNS failure, i/o timeout or connection refused for blocked egress, proxyconnect tcp when the proxy itself is unreachable, and x509: certificate signed by unknown authority when a TLS-intercepting proxy is in the path. The same causes at startup read as Failed to reinitialize target ... Reason: operation error STS: GetCallerIdentity, ....
Fix: allow outbound HTTPS (443) from the Director host to sts.us-east-1.amazonaws.com and to logs.us-east-1.amazonaws.com, substituting your region. Both are needed, because the identity check runs at every initialization. If you use a proxy, set HTTPS_PROXY for the Director service. Loopback and link-local addresses always bypass the proxy, so instance role credentials keep working behind a mandatory proxy. The target has no TLS settings of its own and trusts the operating system certificate store, so install an intercepting proxy's CA certificate at the OS level. Note that endpoint replaces the address for the credential check as well, so a local emulator that does not answer STS calls never gets past startup. Retried until fixed.
"record rejected by target: event size ... exceeds the CloudWatch per-event limit"
Sender worker 1 execute() failed for basic_cloudwatch: record rejected by target: event size 300026 bytes exceeds the CloudWatch per-event limit of 262144 bytes
Sender worker 1 deterministic failure for basic_cloudwatch after 4 attempts — dropping (giving up): record rejected by target: event size 300026 bytes exceeds the CloudWatch per-event limit of 262144 bytes
Cause: one record is too large. The limit is 262,144 bytes per event, and every event carries 26 bytes of overhead on top of its message.
Fix: trim or split the record in a pipeline before it reaches the target, for example by dropping the oversized field. That one record is dropped after 4 deliveries, while the records ahead of it in the same payload are sent and the rest of the traffic keeps flowing. A batch also has a ceiling of 1,048,576 bytes in total: when the next event does not fit, the target sends what it has and starts a new batch, which is normal and is not logged.
"CloudWatch rejected log events in ..." with events missing from the stream
[Warning] [director] [target-<target id>] [basic_cloudwatch] CloudWatch rejected log events in application-logs/app-server-01 (batch size 5000): tooOldEndIndex=312 tooNewStartIndex=-1 expiredEndIndex=-1
Cause: CloudWatch Logs accepted the request and then discarded individual events because of their timestamps. The three numbers say which events were refused: tooOldEndIndex marks events older than 14 days, expiredEndIndex marks events older than the log group's retention setting, and tooNewStartIndex marks events more than 2 hours in the future. A -1 means nothing was refused for that reason. A record whose timestamp could not be parsed becomes 1970-01-01, so a parsing problem in a pipeline also lands here as too old.
How to notice it: the delivery counts as successful, so the connection status stays connected and no error is logged. The warning above is the only trace. Search the Director logs for CloudWatch rejected log events, and watch the target's dropped counter in the stats view. A dropped counter that climbs while the target looks healthy is this.
Fix: correct the timestamps at the source, or add a pipeline that sets @timestamp to the ingest time for feeds that replay old data. Extend the log group's retention if the events are genuinely older than it, and check the clock and time zone of the sending host for events in the future. The events that were refused are dropped silently. They are counted as dropped and are never retried.
The target is healthy but nothing arrives in CloudWatch
Check these in order. Remember first that with interval or cron set, events accumulate until the next tick, so a quiet stream may simply not have been flushed yet.
-
debug.dont_send_logsis enabled. Events are processed and then discarded instead of being sent. The noticeLog sending is disabled for this targetis only written whendebug.statusis also enabled, so withdebug.status: falsethis is completely silent. The Debug Configuration example on this page sets both. Remove the flag. -
The events are being refused for their timestamps. See the entry above, and check the dropped counter.
-
The data went somewhere else. If only one of
keyandsecretis set, the target authenticates as the host's role, which may belong to another account. A valid but wrongregionputs the group and the stream in that region. Both deliver successfully.
"ValidateConfig failed" with a configuration reason
ValidateConfig failed for target "basic_cloudwatch": log_group is required for amazon cloudwatch target
Cause: the target was rejected when the configuration was read. The reason is log_group is required for amazon cloudwatch target, log_stream is required for amazon cloudwatch target, or max_events must be between 1 and 10000, got 20000 with your own value.
Fix: set the missing field, or bring max_events into the range 1 to 10000. Nothing is sent while the configuration is rejected, and Director re-reads it about every 30 seconds, so the target starts as soon as the value is valid. Note that max_events: 0, and any value that is not a number, is accepted and silently becomes the default of 10000.