Amazon Kinesis

Amazon AWS Message Queue

Synopsis

Creates a target that writes log messages to Amazon Kinesis Data Streams with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits. Amazon Kinesis Data Streams is a fully managed streaming data service that enables real-time data processing at scale.

Schema

- name: <string>
  description: <string>
  type: amazonkinesis
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    stream: <string>
    partition_key: <string>
    max_events: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazonkinesis
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom Kinesis endpoint URL (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Stream Configuration

Field	Required	Default	Description
stream	Y	-	Kinesis Data Stream name
partition_key	N	"default"	Partition key for distributing records across shards
max_events	N	500	Maximum number of events per batch (1-500)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

note

Amazon Kinesis Data Streams supports a maximum of 500 records per PutRecords request. The max_events parameter must be between 1 and 500.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Amazon Kinesis Data Streams is a fully managed streaming data service that captures and stores data in real time. This target allows you to send log messages to Kinesis streams for processing by downstream applications.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM Action	Purpose
sts:GetCallerIdentity	Validate credentials at initialization
kinesis:PutRecords	Send batch of records to stream

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "STSIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "KinesisWrite",
      "Effect": "Allow",
      "Action": "kinesis:PutRecords",
      "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME"
    }
  ]
}

Stream and Shard Architecture

Kinesis Data Streams uses shards as the base throughput unit. Each shard provides:

• Write capacity: 1 MB/second or 1,000 records per second
• Read capacity: 2 MB/second

Records are distributed across shards based on the partition key. A well-distributed partition key ensures even load across shards.

Partition Key Strategy

The partition_key parameter determines how records are distributed across shards:

Static Partition Key (default: "default")

• All records go to the same shard
• Simple but can create hot shards
• Suitable for low-volume streams or testing

Dynamic Partition Key

• Use different keys to distribute load
• Records with the same key go to the same shard
• Maintains ordering for records with the same key
• Better performance for high-volume streams

Batch Processing

The target accumulates messages in memory and sends them in batches using the PutRecords API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 500 records per request (Amazon Kinesis limit).

Data Retention

Kinesis Data Streams retains data for 24 hours by default, with the option to extend retention up to 365 days. Data is available for consumption by multiple applications simultaneously.

Encryption

Kinesis automatically encrypts data at rest using AWS KMS. Data in transit is encrypted using TLS. All connections to Kinesis use HTTPS endpoints.

Error Handling

If any record in a batch fails, the whole batch is reported as failed and the entire payload is sent again on the next attempt, including the records that were already accepted. The log line gives the failed and total counts but not the reason for each record. Common failure reasons include:

• Throttling due to exceeding shard limits
• Invalid partition key
• Record size exceeding the stream's per-record limit

See Troubleshooting for the errors these produce and how to resolve them.

Integration with AWS Services

Kinesis Data Streams integrates with other AWS services:

• AWS Lambda for serverless processing
• Amazon Kinesis Data Firehose for delivery to data stores
• Amazon Kinesis Data Analytics for SQL-based stream processing
• Amazon CloudWatch for monitoring and alarms

Examples

Basic Configuration

The minimum configuration for a Kinesis target:

targets:
  - name: basic_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "application-logs"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
  - name: iam_kinesis
    type: amazonkinesis
    properties:
      region: "us-east-1"
      stream: "application-logs"

note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate Kinesis permissions attached.

With Custom Partition Key

Configuration with a custom partition key for better distribution:

targets:
  - name: distributed_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "distributed-logs"
      partition_key: "server-01"

High Throughput

Configuration optimized for high-volume data:

targets:
  - name: high_volume_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "high-volume-logs"
      partition_key: "load-balanced"
      max_events: 500
      timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
  - name: temp_creds_kinesis
    type: amazonkinesis
    properties:
      key: "ASIATEMP1234567890AB"
      secret: "tempSecretKeyExample1234567890"
      session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
      region: "us-west-2"
      stream: "temporary-logs"

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "normalized-logs"
      field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
  - name: reliable_kinesis
    type: amazonkinesis
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "critical-logs"
      max_events: 100

Multiple Regions

Configuration for Kinesis stream in different region:

targets:
  - name: eu_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      stream: "eu-application-logs"
      partition_key: "eu-server"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "scheduled-logs"
      max_events: 500
      interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_kinesis
    type: amazonkinesis
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "test-logs"
      debug:
        status: true
        dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing (e.g., LocalStack):

targets:
  - name: local_kinesis
    type: amazonkinesis
    properties:
      key: "test"
      secret: "test"
      region: "us-east-1"
      endpoint: "http://localhost:4566"
      stream: "local-test-stream"

Production Configuration

Configuration for production with optimal settings:

targets:
  - name: production_kinesis
    type: amazonkinesis
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      stream: "production-logs"
      partition_key: "prod-cluster-01"
      max_events: 500
      timeout: 60
      field_format: "cim"

Troubleshooting

This section covers the errors you are most likely to see with the amazonkinesis target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason:, or after the last colon, is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

This target never marks a record as rejected, so nothing is ever dropped after a fixed number of attempts. A batch that AWS refuses for a deterministic reason, such as a record that is too large, is retried until you change something, and the queue behind it does not move.

Which permission is missing?

Error text	Missing IAM action	Allowed on
api error AccessDeniedException naming kinesis:PutRecords	kinesis:PutRecords	The stream ARN, for example arn:aws:kinesis:us-east-1:000000000000:stream/application-logs
operation error STS: GetCallerIdentity with api error InvalidClientTokenId or api error SignatureDoesNotMatch	sts:GetCallerIdentity	*. An identity policy cannot deny this call, so these codes mean the credentials are wrong rather than under-privileged. A service control policy can still block it
api error KMSAccessDeniedException, or another code beginning with KMS	kms:GenerateDataKey	The customer-managed KMS key that encrypts the stream

The stream itself is a prerequisite rather than a permission. Director never creates a stream and never looks one up, so it must already exist in region before the target starts.

Minimum policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "STSIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "KinesisWrite",
      "Effect": "Allow",
      "Action": "kinesis:PutRecords",
      "Resource": "arn:aws:kinesis:us-east-1:000000000000:stream/application-logs"
    }
  ]
}

Add a third statement allowing kms:GenerateDataKey on the key ARN when the stream is encrypted with a customer-managed key. A stream that uses the AWS managed key needs nothing extra.

"api error SignatureDoesNotMatch" or "api error InvalidClientTokenId"

[Error] [director] [target-<target id>] [production_kinesis] Failed to reinitialize target "production_kinesis" (attempt 6). Reason: operation error STS: GetCallerIdentity, https response error StatusCode: <status>, RequestID: <request id>, api error SignatureDoesNotMatch: ...

Cause: the target calls sts:GetCallerIdentity once at startup to prove the credentials, and AWS refused them. InvalidClientTokenId typically means the access key ID in key does not exist, or was deleted or deactivated. SignatureDoesNotMatch typically means secret does not belong to key. A typo, trailing whitespace from a paste, or a rotated secret all produce it, and so does a badly wrong clock on the Director host. ExpiredToken and ExpiredTokenException mean the temporary token in session has run out, and Director never refreshes it.

Fix: create a fresh access key in IAM and paste both halves again. Check that the host clock is synchronized. If key or secret holds a ${ENV} or $secret{...} token, confirm that the value it resolves to is the one you expect. Replace an expired session token with long-lived credentials or an IAM role, and treat session as something for short tests only.

Nothing is sent while the target fails to start. Data waits in the queue and goes out once the credentials are accepted.

"no EC2 IMDS role found"

[Error] [director] [target-<target id>] [production_kinesis] Failed to reinitialize target "production_kinesis" (attempt 9). Reason: operation error STS: GetCallerIdentity, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, ...

Cause: no static credentials were found, so the target fell back to the AWS default credential chain, which ended at the instance metadata service and got nothing back. Static credentials are used only when key and secret are both set. Filling in one and leaving the other empty is treated exactly like setting neither, and the error never mentions the half you did fill in. A ${ENV} or $secret{...} token that resolves to an empty value counts as empty.

Fix: set both key and secret, or set neither and attach an IAM role to the EC2 instance, ECS task or EKS pod that runs Director. When you rely on a role, 169.254.169.254 must be reachable from the host. That address is always contacted directly, so a mandatory proxy does not break it.

Nothing is sent while the target fails to start.

warning

On a host that does have an instance role, a half-configured pair does not fail at all. The target sends under the role instead of under the key you configured, which can be a different account, and nothing in the log says so. If records arrive under an identity you did not expect, check that both key and secret are set.

"failed to resolve access key" or "failed to resolve secret key"

[Error] [director] [target-<target id>] [production_kinesis] Failed to reinitialize target "production_kinesis" (attempt 4). Reason: failed to resolve access key: credential: env variable "AWS_ACCESS_KEY_ID" is not set

Cause: key, secret or session holds a token that Director could not resolve. The inner text names the problem. Expect credential: env variable ... is not set, credential: store ... not found in configuration, credential: failed to parse token ..., or credential: no credentials configured.

Fix: for a ${ENV} token, export the variable for the Director service itself, not only for your shell. The service picks up a new environment when it restarts. For a $secret{...} token, check that the store named in the token exists in the configuration, that the reference is spelled correctly, and that the store answers within 30 seconds.

Nothing is sent while the token cannot be resolved. Data waits in the queue.

"api error AccessDeniedException" on PutRecords

[Error] [director] [target-<target id>] [production_kinesis] Sender worker 1 Finalize failed on flush for target "production_kinesis": failed to put records to kinesis: operation error Kinesis: PutRecords, https response error StatusCode: <status>, RequestID: <request id>, api error AccessDeniedException: ...

Cause: the credentials are valid, which is why the target started, but the identity may not call kinesis:PutRecords on this stream. The message typically names the identity and the stream ARN it was refused on. A permission boundary, a service control policy, or a resource policy on the stream produces the same code.

Fix: attach the policy above to the user or role named in the message. Then compare the stream ARN in the message with the ARN of your stream. A different account id or region in that ARN means the target is pointed somewhere you did not intend.

The batch is retried until the permission is in place. Nothing is lost, and the queue grows meanwhile.

"failed to put 37/500 records to kinesis"

[Error] [director] [target-<target id>] [production_kinesis] Sender worker 4 Finalize failed on flush for target "production_kinesis": failed to put 37/500 records to kinesis

Symptom: the call itself succeeded. AWS accepted the request and then refused some of the records inside it. The first number is how many records were refused, the second is how many were in the batch. Per-shard throttling is the usual cause.

Cause: partition_key decides which shard a record is written to, and all records carrying the same key go to the same shard. The default is the single constant value "default", so the whole target writes to one shard. That shard accepts 1 MB per second and 1,000 records per second. Anything above that is refused record by record while the other shards stay idle, which is why the stream as a whole still looks far below its limit.

What the log shows: the two counts, and nothing else. The per-record error code is not written to the log, so the line does not tell you which records were refused, on which shard, or for which reason. What you can check is the target's own statistics, and the incoming records per shard in the AWS console. A single shard carrying the entire load confirms the partition key as the cause.

Fix:

1. Give the load more than one key. A partition key that varies from record to record is what spreads records across shards. partition_key is one fixed value per target, so splitting the flow across several targets, each with its own partition_key, is how you vary it in practice.
2. Give the stream more capacity. Add shards, or switch the stream to on-demand capacity.
3. Send less at once. Lower max_events, or set interval so batches leave on a schedule. See Scheduling and Pool Behavior.

The whole payload is sent again on the next attempt, including the records AWS already accepted, and this repeats until one batch succeeds in full. Expect duplicates in the stream, and expect the delivered counter to count the accepted records again on every attempt. De-duplicate on a record identifier downstream if your consumer cannot tolerate that.

"api error ProvisionedThroughputExceededException"

[Error] [director] [target-<target id>] [production_kinesis] Sender worker 4 Finalize failed on flush for target "production_kinesis": failed to put records to kinesis: operation error Kinesis: PutRecords, exceeded maximum number of attempts, 3, https response error StatusCode: <status>, RequestID: <request id>, api error ProvisionedThroughputExceededException: ...

Cause: the whole request was throttled rather than individual records. The message typically names the shard. exceeded maximum number of attempts, 3 means the call was already retried three times inside the target's timeout window before the error was reported.

Fix: the three fixes in the entry above apply unchanged. Raising timeout to 60 also gives those internal retries more room when the throttling is short-lived, at the cost of holding the batch longer.

The batch is retried until it is accepted.

"api error ResourceNotFoundException" or "Invalid Configuration: Missing Region"

[Error] [director] [target-<target id>] [production_kinesis] Sender worker 2 execute() failed for production_kinesis: target broken: failed to finalize target cache: failed to put records to kinesis: operation error Kinesis: PutRecords, https response error StatusCode: <status>, RequestID: <request id>, api error ResourceNotFoundException: ...

Cause: ResourceNotFoundException means the stream named in stream does not exist in region for the account the credentials belong to. The message typically names the stream and the account it looked in. A deleted stream, a misspelled name, and a stream that lives in another region all look the same. endpoint rule error, Invalid Configuration: Missing Region at startup means region is empty and the host supplies no region either, so there is no hostname to call.

Fix: set stream to the stream name, not its ARN. Set region explicitly on the target rather than relying on the host environment. Create the stream if it does not exist, because Director never creates one.

The payload is redelivered until the stream is reachable. Nothing is sent at all while the region is missing.

"api error ValidationException" or "api error InvalidArgumentException"

[Error] [director] [target-<target id>] [production_kinesis] Sender worker 2 execute() failed for production_kinesis: target broken: failed to finalize target cache: failed to put records to kinesis: operation error Kinesis: PutRecords, https response error StatusCode: <status>, RequestID: <request id>, api error ValidationException: ...

Cause: AWS rejected the request as malformed rather than unauthorized. The message typically names the constraint and the position of the offending record in the batch. Three limits account for most of these.

• One record, counting its data and its partition key together, is over the stream's per-record size limit. The target sends records of any size and does not check this first. Read the limit from the message rather than assuming a figure, because it depends on the stream.
• The batch as a whole is over the per-request size limit. 500 large records can exceed it even when each record on its own is acceptable.
• partition_key is longer than 256 characters, or the stream name contains characters Kinesis does not accept. These typically come back as InvalidArgumentException.

Fix: trim or split oversized events in the pipeline before they reach the target. Lower max_events so a batch of large records stays under the request limit. Shorten partition_key.

warning

This is the failure that stops the queue. The rejection is deterministic, so every retry fails identically and the batch is retried every few seconds indefinitely. Nothing queued behind it is delivered until the oversized event is out of the queue and the pipeline has stopped producing more.

"no such host", "context deadline exceeded", or a certificate error

[Error] [director] [target-<target id>] [production_kinesis] Failed to reinitialize target "production_kinesis" (attempt 3). Reason: operation error STS: GetCallerIdentity, exceeded maximum number of attempts, 3, request send failed, Post "https://sts.us-east-1.amazonaws.com/": dial tcp: lookup sts.us-east-1.amazonaws.com: no such host

Cause: the tail of the line says which one it is.

• no such host means the hostname does not resolve. The hostname is built from region, so a misspelled region produces this just as reliably as a DNS outage does.
• connection refused, i/o timeout, or proxyconnect tcp: ... means the host cannot reach AWS, or the proxy is down.
• x509: certificate signed by unknown authority means TLS is being intercepted and a private CA is presented. This target has no custom CA setting and no option to skip verification.
• context deadline exceeded means the call did not finish inside timeout, which is 30 seconds by default.

Fix: allow outbound TCP 443 from the Director host to kinesis.<region>.amazonaws.com and to sts.<region>.amazonaws.com. Set HTTP_PROXY, HTTPS_PROXY and NO_PROXY for the Director service when egress goes through a proxy. For an intercepting proxy, install its CA in the host trust store or exempt the AWS hostnames from interception. For context deadline exceeded, raise timeout to 60 and lower max_events.

Everything here is retried until the connection works.

note

endpoint replaces the base endpoint for the Kinesis calls and for the sts:GetCallerIdentity call at startup. An endpoint that serves only Kinesis, such as a Kinesis interface endpoint in a VPC, makes startup fail. Leave endpoint empty unless you are pointing the target at a local emulator that answers both.

"stream is required for amazon kinesis target" or "max_events must be between 1 and 500"

[Error] [director] [target-<target id>] [production_kinesis] ValidateConfig failed for target "production_kinesis": max_events must be between 1 and 500, got 1000

Cause: the configuration was rejected before the target started. Either stream is empty, or max_events is above 500 or negative. 500 records is the Kinesis limit for one PutRecords call.

Fix: set stream, and keep max_events between 1 and 500.

Nothing is sent while the configuration is invalid. The check runs again about every 30 seconds, so correcting the configuration is enough. Note that max_events: 0 is read as "not set" and becomes the default of 500 instead of failing this check, and timeout: 0 becomes 30 seconds the same way.

The target is connected but nothing arrives in the stream

1. debug.dont_send_logs is enabled. Events are processed by the pipeline and then discarded instead of being sent, and the target still reports as healthy. The only notice is one startup line, Log sending is disabled for this target (production_kinesis). Logs will be processed by the pipeline but will not be sent to the target., and it is written only when debug.status is also true. With debug.status off, the drop is silent. Remove dont_send_logs. It is a testing flag.
2. The records went to a stream of the same name in another region. They are written there and nothing is logged. Check region against the region in the stream ARN, not against the stream name.
3. field_format could not normalize an event. That record is sent as it was, without normalization and without a log line. Test the normalization in the pipeline before relying on it here.