Amazon CloudWatch Logs

Amazon AWS Logging

Synopsis

Creates a target that writes log messages to Amazon CloudWatch Logs with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits.

Schema

- name: <string>
  description: <string>
  type: amazoncloudwatch
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    log_group: <string>
    log_stream: <string>
    max_events: <numeric>
    timeout: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazoncloudwatch
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom CloudWatch Logs endpoint URL (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Log Configuration

Field	Required	Default	Description
log_group	Y	-	CloudWatch Logs log group name
log_stream	Y	-	CloudWatch Logs log stream name
max_events	N	10000	Maximum number of events per batch (1-10000)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

note

Amazon CloudWatch Logs supports a maximum of 10,000 log events per PutLogEvents request. The max_events parameter must be between 1 and 10,000.

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

Log Groups and Streams

CloudWatch Logs organizes log data into log groups and log streams:

Log Groups

• Container for log streams
• Define retention, monitoring, and access control settings
• Created automatically by the target if it doesn't exist

Log Streams

• Sequence of log events from the same source
• Events must be in chronological order within a stream
• Created automatically by the target if it doesn't exist

Batch Processing

The target accumulates messages in memory and sends them in batches using the PutLogEvents API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10,000 log events per request.

Sequence Tokens

CloudWatch Logs uses sequence tokens to ensure log events are written in order. The target automatically manages sequence tokens for each batch, maintaining the correct order of log events within the log stream.

Timestamps

Log events are sent with timestamps in milliseconds. The target automatically converts the event epoch timestamp from nanoseconds to milliseconds as required by CloudWatch Logs.

Data Retention

CloudWatch Logs retains log data indefinitely by default. You can configure retention periods from 1 day to 10 years at the log group level through the AWS Console or API.

Encryption

CloudWatch Logs encrypts log data at rest by default. All connections to CloudWatch Logs use HTTPS endpoints for encryption in transit.

Error Handling

The target handles common CloudWatch Logs errors:

• Automatically creates log groups and log streams if they don't exist
• Manages sequence tokens for ordered log delivery
• Handles ResourceAlreadyExistsException when creating existing resources

Examples

Basic Configuration

The minimum configuration for a CloudWatch Logs target:

targets:
  - name: basic_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "application-logs"
      log_stream: "app-server-01"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
  - name: iam_cloudwatch
    type: amazoncloudwatch
    properties:
      region: "us-east-1"
      log_group: "application-logs"
      log_stream: "app-server-01"

note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate CloudWatch Logs permissions attached.

High Volume Logs

Configuration optimized for high-volume data:

targets:
  - name: high_volume_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "high-volume-logs"
      log_stream: "data-pipeline"
      max_events: 10000
      timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
  - name: temp_creds_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "ASIATEMP1234567890AB"
      secret: "tempSecretKeyExample1234567890"
      session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
      region: "us-west-2"
      log_group: "temporary-logs"
      log_stream: "session-logs"

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "normalized-logs"
      log_stream: "structured-events"
      field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
  - name: reliable_cloudwatch
    type: amazoncloudwatch
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "critical-logs"
      log_stream: "production-events"
      max_events: 1000

Multiple Applications

Configuration for different application log streams:

targets:
  - name: web_server_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "application-logs"
      log_stream: "web-server"

  - name: api_server_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "application-logs"
      log_stream: "api-server"

Multiple Regions

Configuration for CloudWatch Logs in different region:

targets:
  - name: eu_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      log_group: "eu-application-logs"
      log_stream: "eu-server-01"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "scheduled-logs"
      log_stream: "batch-events"
      max_events: 5000
      interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "test-logs"
      log_stream: "debug-stream"
      debug:
        status: true
        dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing:

targets:
  - name: local_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "test"
      secret: "test"
      region: "us-east-1"
      endpoint: "http://localhost:4566"
      log_group: "local-test-logs"
      log_stream: "local-stream"

Production Configuration

Configuration for production with optimal settings:

targets:
  - name: production_cloudwatch
    type: amazoncloudwatch
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "/aws/application/production"
      log_stream: "app-cluster-01"
      max_events: 10000
      timeout: 60
      field_format: "cim"

Container Logs

Configuration for containerized application logs:

targets:
  - name: container_cloudwatch
    type: amazoncloudwatch
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      log_group: "/ecs/application"
      log_stream: "container-instance-01"
      max_events: 8000