Amazon S3

Amazon AWS Long Term Storage

Synopsis

Creates a target that writes log messages to Amazon S3 buckets with support for various file formats, authentication methods, and multipart uploads. The target handles large file uploads efficiently with configurable rotation based on size or event count.

Schema

- name: <string>
  description: <string>
  type: awss3
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    use_path_style: <boolean>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be awss3
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom S3-compatible endpoint URL (for non-Amazon S3 services)
use_path_style	N	computed	Use path-style S3 addressing. Derived from endpoint when unset: false for an empty endpoint or an amazonaws.com host, true for any other endpoint — so a custom S3-compatible endpoint (MinIO, LocalStack, Ceph) gets path-style without configuration. Set it explicitly to override

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Connection

Field	Required	Default	Description
part_size	N	5	Multipart upload part size in megabytes (minimum 5MB)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
bucket	N*	-	Default S3 bucket name (acts as catch-all when buckets is also specified)
buckets	N*	-	Array of bucket configurations for file distribution
buckets.bucket	Y	-	S3 bucket name
buckets.name	Y	-	File name template
buckets.format	N	"json"	Output format: json, multijson, avro, parquet
buckets.compression	N	-	Compression algorithm (e.g., zstd, gzip, lz4, snappy). See the File Formats include below
buckets.extension	N	Matches format	File extension override
buckets.schema	N*	-	Schema reference (required for Avro and Parquet formats)
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	Default file name template (used with bucket for catch-all)
format	N	"json"	Default output format: json, multijson, avro, parquet (used with bucket for catch-all)
compression	N	zstd	Default compression (used with bucket for catch-all)
extension	N	Matches format	Default file extension (used with bucket for catch-all)
schema	N	-	Default schema reference (used with bucket for catch-all)
max_size	N	33554432	Maximum file size in bytes before rotation (32 MB)
batch_size	N	100000	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to S3 and a new file is created. Setting max_size to 0 does not disable rotation — an explicit 0 is replaced by the 32 MB default (MustInt64), so there is no unlimited setting. Raise the value instead.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Amazon S3 target supports writing to different buckets with various file formats and schemas, and uploads each rotated file to the bucket it is routed to.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods are validated against the storage service at initialization, before any data is sent. With a top-level bucket the target calls HeadBucket on that bucket. When only buckets is configured, and for every S3-compatible service, it calls ListBuckets instead.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM Action	Purpose
s3:ListBucket	Probe the bucket named in bucket at initialization
s3:ListAllMyBuckets	Probe the account at initialization, when only buckets is configured or the endpoint is an S3-compatible service
s3:PutObject	Upload log files to bucket (also covers multipart upload lifecycle)
s3:AbortMultipartUpload	Abort failed multipart uploads

note

Per the AWS Service Authorization Reference, s3:PutObject covers CreateMultipartUpload, UploadPart, and CompleteMultipartUpload. Only s3:AbortMultipartUpload requires its own IAM action.

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketProbe",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET_NAME"
    },
    {
      "Sid": "AccountProbe",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Sid": "S3Upload",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

note

The S3 upload manager automatically switches between single-part PutObject and multipart upload based on the part_size configuration.

File Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	zstd	deflate, snappy, zstd
Parquet	zstd	gzip, snappy, zstd, brotli, lz4

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

Bucket Routing

The target supports flexible bucket routing through pipeline configuration or explicit bucket settings:

Configuration-based routing: Define multiple buckets in the target configuration, each with its own format, compression, and schema settings. Logs are routed to specific buckets based on configuration.

Pipeline-based routing: Use the bucket field in pipeline processors to dynamically route logs to different buckets at runtime. This enables conditional routing based on log content, source, or other attributes.

Catch-all routing: When a log doesn't match any specific bucket configuration or when no bucket field is set in the pipeline, logs are routed to the catch-all bucket (configured via the bucket field in target properties).

Routing priority:

1. Pipeline bucket field (highest priority)
2. Configured buckets in buckets array (if bucket name matches)
3. Default bucket field (catch-all, lowest priority)

This multi-level routing enables flexible data distribution strategies, such as routing different log types to different buckets based on content analysis, source system, severity level, or any other runtime decision.

Templates

The following template variables can be used in file names:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	awss3
{{.Table}}	Bucket name	logs
{{.Thread}}	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	3
{{.ServiceRoot}}	The service root directory	/opt/vmetric

Multipart Upload

Large files automatically use S3 multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple S3 buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require a schema. The schema value can be a Library schema name, a built-in schema name, or an inline JSON definition. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup, so an Avro schema must be a name or inline JSON. See Avro and Parquet for the JSON definition format.

Examples

Basic Configuration

The minimum configuration for a JSON S3 target:

targets:
  - name: basic_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "datastream-logs"

Pipeline-Based Routing

Dynamic bucket routing using pipeline processors to analyze log content and route to appropriate buckets:

targets:
  - name: smart_routing_s3
    type: awss3
    pipelines:
      - dynamic_routing
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      buckets:
        - bucket: "security-events"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-events"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "system-events"
          name: "system-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "other-events"
      name: "other-{{.Timestamp}}.json"
      format: "json"

pipelines:
  - name: dynamic_routing
    processors:
      - set:
          field: "_vmetric.bucket"
          value: "security-events"
          if: "ctx.event_type == 'security'"
      - set:
          field: "_vmetric.bucket"
          value: "application-events"
          if: "ctx.event_type == 'application'"
      - set:
          field: "_vmetric.bucket"
          value: "system-events"
          if: "ctx.event_type == 'system'"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Multiple Buckets with Different Formats

Configuration for distributing data across multiple S3 buckets with different formats:

targets:
  - name: multi_bucket_export
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-west-2"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_s3
    type: awss3
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Troubleshooting

This section covers the errors you are most likely to see with the awss3 target, what causes each one, and how to fix it. The same behavior applies to the S3-compatible services documented on their own pages: Amazon Security Lake, Cloudflare R2, IBM Cloud Object Storage, MinIO, DigitalOcean Spaces, Alibaba Cloud OSS, Oracle Cloud Object Storage, Backblaze B2, Scaleway Object Storage, and Wasabi. Where one of those behaves differently from Amazon S3, the entry says so.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission is missing?

Before it sends anything, the target makes one probe call, and that call decides which permission you need. awss3 with a top-level bucket is probed with HeadBucket on that bucket. Everything else is probed with ListBuckets, which is an account-wide call. That covers awss3 configured with only buckets, every S3-compatible service, and Amazon Security Lake, which is always configured with a bucket list and so never takes the single-bucket path. Match the error you see against this table.

Error text	Missing IAM action	Must be allowed on
api error Forbidden: Forbidden after HeadBucket	s3:ListBucket	arn:aws:s3:::my-bucket, the bucket itself, not its objects
api error AccessDenied after ListBuckets	s3:ListAllMyBuckets	*, account-wide
api error AccessDenied after PutObject, CreateMultipartUpload, or UploadPart	s3:PutObject	arn:aws:s3:::my-bucket/*, for every bucket in bucket and in buckets
A multipart upload fails and leaves parts behind	s3:AbortMultipartUpload	arn:aws:s3:::my-bucket/*
api error AccessDenied after PutObject into a bucket whose default encryption is SSE-KMS	kms:GenerateDataKey and kms:Decrypt, typically	The key that encrypts the bucket

A policy that covers both probes and the upload:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "BucketProbe", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-bucket" },
    { "Sid": "AccountProbe", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" },
    { "Sid": "Upload", "Effect": "Allow", "Action": ["s3:PutObject", "s3:AbortMultipartUpload"], "Resource": "arn:aws:s3:::my-bucket/*" }
  ]
}

AccountProbe is only needed on the ListBuckets path, so with awss3 and a top-level bucket you can drop that statement. For a KMS-encrypted bucket, add the key as well, for example arn:aws:kms:us-east-1:000000000000:key/KEY_ID.

warning

Only the top-level bucket is probed at startup. Bucket names under buckets are not checked until the first upload is routed to them, so a typo there passes startup and fails later.

"api error Forbidden: Forbidden"

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 3). Reason: operation error S3: HeadBucket, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error Forbidden: Forbidden

Cause: the bucket probe was refused. The call returns an empty body, so one message covers three different problems: the key and secret pair is wrong, the identity is missing s3:ListBucket on the bucket, or the bucket belongs to a different account.

Fix: check those three in order. Confirm key and secret are the pair you meant to use, with no whitespace pasted along with them. Grant s3:ListBucket on arn:aws:s3:::my-bucket to that identity. Then confirm the bucket is owned by the account the key belongs to, remembering that cross-account access also needs a bucket policy that allows the identity.

Nothing is sent while this lasts. Incoming data stays queued and is retried until you fix the cause.

"api error AccessDenied" after ListBuckets

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 6). Reason: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: ...

Cause: the target was probed with ListBuckets, and the credentials may not list the buckets of the account. That call needs s3:ListAllMyBuckets on *, which a policy written for a single bucket does not grant. A credential scoped to one bucket typically fails here even though it can write objects without any trouble.

Fix: grant the account-wide action, or avoid the call.

• On Amazon S3, grant s3:ListAllMyBuckets on *. Adding a top-level bucket also works, because the target is then probed with HeadBucket instead.
• On Cloudflare R2, use an account-scoped API token rather than a bucket-scoped one.
• On MinIO, and other services with their own policy language, add s3:ListAllMyBuckets to the user's policy.

Nothing is sent while this lasts, and it is retried until you fix the cause.

"InvalidAccessKeyId", "SignatureDoesNotMatch", or an expired token

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 2). Reason: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error SignatureDoesNotMatch: ...

Cause: unlike the bucket probe, ListBuckets returns a body, so the error code names the problem. The codes you will typically see:

Code	What it means
InvalidAccessKeyId	The access key does not exist on this service. A key issued by another provider, or for another account, looks like this
SignatureDoesNotMatch	The secret is wrong, or region does not match the region the endpoint serves. Whitespace pasted with the secret is a common cause
ExpiredToken or InvalidClientTokenId	The session token has expired, or it does not belong to this key pair
RequestTimeTooSkewed	The Director host clock is more than 15 minutes off. Fix NTP on the host

Fix: re-issue the key pair in the provider's console and paste both values again. For temporary credentials, take key, secret, and session from the same set, and prefer an IAM role or a $secret{...} reference over pasting a token that expires. Nothing is sent while this lasts, and it is retried until you fix the cause.

"no EC2 IMDS role found" or "failed to resolve access key"

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 1). Reason: operation error S3: HeadBucket, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, request canceled, context deadline exceeded

Cause: key and secret are used only when both resolve to a non-empty value. If either one is empty, both are ignored and the target falls back to the default AWS credential chain: environment variables, the shared AWS config file, then the instance role of the host. On a host with none of those, the chain ends at the instance metadata service and reports this. A ${VAR} or $secret{...} reference that resolves to an empty string produces the same fallback while the configuration still looks complete. When the reference itself cannot be resolved, you get the more direct failed to resolve access key: credential: env variable "AWS_ACCESS_KEY_ID" is not set instead, or credential: store "..." not found in configuration for a secret store.

Fix: set both key and secret, or run Director on an AWS instance whose role carries the permissions above. On every service other than Amazon S3 the fallback can never succeed, so both fields are required there in practice. Export environment variables for the Director service itself, not only in the shell you tested from, and restart the service so it picks them up. Nothing is sent while this lasts, and it is retried until you fix the cause.

"A region must be set", "Invalid region", or "Custom endpoint ... was not a valid URI"

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 1). Reason: operation error S3: ListBuckets, failed to resolve service endpoint, endpoint rule error, A region must be set when sending requests to S3.

Cause: the connection settings could not be turned into a request. A region must be set when sending requests to S3. means region is empty and none could be read from the environment or the shared AWS config on the host. Invalid region: region was not a valid DNS name. means region holds something that is not a region code, such as a console display name, or a value with spaces, underscores, or capitals. The Custom endpoint message means endpoint is not a URL.

Fix: set region to the region code your provider publishes, not the name shown in its console. Services that have no AWS-style regions still need a value: Cloudflare R2 expects auto, and MinIO accepts any value, commonly us-east-1. Set endpoint to a full URL with a scheme, https://host or https://host:port, and use http:// only inside a trusted network. Nothing is sent while this lasts, and it is retried until you fix the cause.

warning

region and endpoint must agree. Requests are signed with region, so an endpoint that serves a different region typically answers with SignatureDoesNotMatch or AuthorizationHeaderMalformed even though each value looks correct on its own.

"api error NotFound: Not Found", or a "NoSuchBucket" error on the first upload

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 2). Reason: operation error S3: HeadBucket, https response error StatusCode: 404, RequestID: ..., HostID: ..., api error NotFound: Not Found

[Error] [director] [target-<target id>] [multi_bucket_routing] Sender worker 1 execute() failed for multi_bucket_routing: target broken: failed to finalize target cache: operation error S3: PutObject, https response error StatusCode: 404, RequestID: ..., HostID: ..., api error NoSuchBucket: ...

Cause: the bucket does not exist under that name in that account, and the target never creates buckets. The first message is the startup probe of the top-level bucket. The second appears when a name under buckets is wrong, because those entries are not touched until the first upload is routed to them. A 400 with api error BadRequest: Bad Request on the same probe typically means the opposite: the bucket exists, but it is in a different region than region.

Fix: correct the name or create the bucket, and check every entry under buckets, not only the catch-all. For the 400, set region to the bucket's own region, because the target does not follow a region redirect for you. Nothing is lost: the batch is discarded from memory, the payloads stay queued, and they are retried until you fix the cause.

"no such host", "connection refused", or "certificate signed by unknown authority"

[Error] [director] [target-<target id>] [basic_s3] Failed to reinitialize target "basic_s3" (attempt 2). Reason: operation error S3: ListBuckets, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://storage.example.local:9000/": dial tcp: lookup storage.example.local: no such host

Cause: Director could not reach the endpoint. The text after request send failed says which step failed.

• no such host means the endpoint hostname does not resolve. It also appears when use_path_style is set to false against a service with no wildcard DNS, because the request then goes to the bucket name as a subdomain of the endpoint host.
• connection refused or i/o timeout means the port is wrong, a firewall is in the way, or the service is down. If HTTPS_PROXY is set for the Director service, an internal host is sent through the proxy unless it is listed in NO_PROXY. Loopback and link-local addresses always go direct.
• tls: failed to verify certificate: x509: certificate signed by unknown authority means the endpoint presents a certificate from a private CA. This target reads no TLS options, so there is no setting to trust a custom CA or to skip verification.

Fix: correct endpoint, open the port, or add the host to NO_PROXY. Leave use_path_style unset so a custom endpoint gets path-style addressing on its own. For a private CA, install the CA certificate in the Director host's own trust store. Nothing is sent while this lasts, and it is retried until you fix the cause.

An endpoint that accepts the connection and then never answers is the worst case. The startup probe has no deadline of its own, so it waits for the operating system's TCP timeout before the error appears at all.

"canceled, context deadline exceeded" while uploading

[Error] [director] [target-<target id>] [basic_s3] Sender worker 0 Finalize failed on flush for target "basic_s3": operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded

Cause: timeout bounds the whole upload, not one part of it, and it defaults to 30 seconds. An object near the 32 MB max_size default does not finish within 30 seconds on a slow, distant, or proxied link. Each retry starts again from the first byte, so it fails in exactly the same way.

Fix: raise timeout to 120 or more, lower max_size so each object is smaller, or raise part_size so fewer parts are uploaded. Any one of the three can be enough. Nothing is lost, and the batch is retried until you fix the cause, but the target makes no progress at all until one of those values changes.

"api error SlowDown" and other throttling

[Error] [director] [target-<target id>] [basic_s3] Sender worker 2 execute() failed for basic_s3: target broken: failed to finalize target cache: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: ..., HostID: ..., api error SlowDown: ...

Cause: too many objects per second against one prefix. exceeded maximum number of attempts, 3 means the request was already retried three times with backoff before the error reached the log. Small objects make this worse, because a new object is uploaded every time batch_size (100000 records) or max_size (32 MB) is reached, and each writer thread rotates on its own.

Fix: raise batch_size and max_size so each object carries more. Set interval or cron so the target flushes on a schedule with a single writer instead of after every payload, as described under Scheduling and Pool Behavior. Spread the keys over more prefixes by putting {{.Year}}/{{.Month}}/{{.Day}} at the front of name. Nothing is lost, and the batch is retried until the service accepts it.

"file holder not found"

[Error] [director] [target-<target id>] [multi_bucket_routing] Sender worker 0 execute() failed for multi_bucket_routing: target broken: failed to send log record: file holder not found

Cause: the target has buckets but no top-level bucket, and a record arrived whose _vmetric.bucket is empty or names a bucket that is not in the list. There is nowhere to write it.

Fix: add a top-level bucket as the catch-all, or make the pipeline set _vmetric.bucket on every record to one of the listed buckets. Nothing is lost, but the payload is redelivered every few seconds and fails the same way each time, which holds up everything queued behind it, so treat this one as urgent.

"no valid buckets found" and other configuration mistakes

These reach you through the same reinitialize loop as a connection failure, so the target keeps retrying a configuration that cannot work until you change it. Nothing is sent while any of them lasts.

Reason text	Fix
no valid buckets found	Set bucket, or at least one entry under buckets
invalid bucket configuration: bucket is required	Every entry under buckets needs a bucket
invalid bucket configuration: name is required	Every entry under buckets needs its own name. Only the top-level name has a default
invalid bucket configuration: schema is required for parquet format	Add schema wherever format is avro or parquet, including entries that inherit the format from the top level
invalid schema format: invalid field format: ...	The schema value was not found in the schema library, so it was read as an inline field:type list and failed on the first token. Check the spelling, and that the schema file is deployed
exceeded total allowed S3 limit MaxUploadParts (10000). Adjust PartSize to fit in this limit	max_size is more than 10000 times part_size. Raise part_size

The target is healthy but the bucket stays empty, or holds only the last batch

Nothing fails here, so there is no error to search for. In the first two cases the data is dropped silently and no counter moves. Check the following in order.

1. The object name has no unique part. A name such as logs-{{.Year}}-{{.Month}}-{{.Day}}.json produces the same object key on every rotation, and each upload replaces the one before it. Only the last batch of the day survives, while the statistics count every batch as delivered. Put {{.Timestamp}} in name, and in the name of every entry under buckets.

2. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No counter moves, no object is written, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well. Remove the flag when you have finished testing.

   Log sending is disabled for this target (basic_s3). Logs will be processed by the pipeline but will not be sent to the target.

3. The records landed in the catch-all bucket. When _vmetric.bucket names a bucket that is not under buckets and a top-level bucket exists, the record is written to the catch-all with no warning. Look there before assuming the data is gone.

4. The objects are not in the format the name claims. A format value that is not recognized falls back to JSON lines, while extension keeps the value you typed, so the objects are named for a format they do not contain. Use json, jsonl, multijson, avro, parquet, or raw, in lower case.

5. The objects are not compressed. For the line formats only gzip is applied. Any other value, including the default zstd, uploads uncompressed while {{.Compression}} in the name still says otherwise. Set compression: gzip for line formats. avro and parquet use their own codecs and are unaffected.

6. Objects are larger than max_size suggests. The limit is measured on the buffer after encoding and compression, and Parquet holds rows back until a row group is complete, so an object can overshoot before it is written out.