Amazon S3

Amazon AWS Long Term Storage

Synopsis

Creates a target that writes log messages to Amazon S3 buckets with support for various file formats, authentication methods, and multipart uploads. The target handles large file uploads efficiently with configurable rotation based on size or event count.

Schema

- name: <string>
  description: <string>
  type: awss3
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    use_path_style: <boolean>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `awss3`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
`key`	N*	-	AWS access key ID for authentication
`secret`	N*	-	AWS secret access key for authentication
`session`	N	-	Optional session token for temporary credentials
`region`	Y	-	AWS region (e.g., `us-east-1`, `eu-west-1`)
`endpoint`	N	-	Custom S3-compatible endpoint URL (for non-Amazon S3 services)
`use_path_style`	N	`false`	Use path-style S3 addressing; required for S3-compatible endpoints without virtual-host DNS (e.g., MinIO, LocalStack)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Connection

Field	Required	Default	Description
`part_size`	N	`5`	Multipart upload part size in megabytes (minimum 5MB)
`timeout`	N	`30`	Connection timeout in seconds
`field_format`	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
`bucket`	N*	-	Default S3 bucket name (acts as catch-all when `buckets` is also specified)
`buckets`	N*	-	Array of bucket configurations for file distribution
`buckets.bucket`	Y	-	S3 bucket name
`buckets.name`	Y	-	File name template
`buckets.format`	N	`"json"`	Output format: `json`, `multijson`, `avro`, `parquet`
`buckets.compression`	N	-	Compression algorithm (e.g., `zstd`, `gzip`, `lz4`, `snappy`). See the File Formats include below
`buckets.extension`	N	Matches `format`	File extension override
`buckets.schema`	N*	-	Schema reference (required for Avro and Parquet formats)
`name`	N	`"vmetric.{{.Timestamp}}.{{.Extension}}"`	Default file name template (used with `bucket` for catch-all)
`format`	N	`"json"`	Default output format: `json`, `multijson`, `avro`, `parquet` (used with `bucket` for catch-all)
`compression`	N	-	Default compression (used with `bucket` for catch-all)
`extension`	N	Matches `format`	Default file extension (used with `bucket` for catch-all)
`schema`	N	-	Default schema reference (used with `bucket` for catch-all)
`max_size`	N	`33554432`	Maximum file size in bytes before rotation (32 MB)
`batch_size`	N	`100000`	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to S3 and a new file is created. For unlimited file size, set the field to 0.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
`debug.status`	N	`false`	Enable debug logging
`debug.dont_send_logs`	N	`false`	Process logs but don't send to target (testing)

Details

The Amazon S3 target supports writing to different buckets with various file formats and schemas. The target provides enterprise-grade cloud storage integration with comprehensive file format support.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

All authentication methods call sts:GetCallerIdentity during initialization to validate credentials before proceeding.

IAM Permissions

When using IAM role-based authentication, the following permissions are required:

IAM Action	Purpose
`sts:GetCallerIdentity`	Validate credentials at initialization
`s3:PutObject`	Upload log files to bucket (also covers multipart upload lifecycle)
`s3:AbortMultipartUpload`	Abort failed multipart uploads

note

Per the AWS Service Authorization Reference, s3:PutObject covers CreateMultipartUpload, UploadPart, and CompleteMultipartUpload. Only s3:AbortMultipartUpload requires its own IAM action.

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "STSIdentity",
      "Effect": "Allow",
      "Action": "sts:GetCallerIdentity",
      "Resource": "*"
    },
    {
      "Sid": "S3Upload",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

note

The S3 upload manager automatically switches between single-part PutObject and multipart upload based on the part_size configuration.

File Formats

Format	Description
`json`	Each log entry is written as a separate JSON line (JSONL format)
`multijson`	All log entries are written as a single JSON array
`avro`	Apache Avro format with schema
`parquet`	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	`zstd`	`deflate`, `snappy`, `zstd`
Parquet	`zstd`	`gzip`, `snappy`, `zstd`, `brotli`, `lz4`

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

Bucket Routing

The target supports flexible bucket routing through pipeline configuration or explicit bucket settings:

Configuration-based routing: Define multiple buckets in the target configuration, each with its own format, compression, and schema settings. Logs are routed to specific buckets based on configuration.

Pipeline-based routing: Use the bucket field in pipeline processors to dynamically route logs to different buckets at runtime. This enables conditional routing based on log content, source, or other attributes.

Catch-all routing: When a log doesn't match any specific bucket configuration or when no bucket field is set in the pipeline, logs are routed to the catch-all bucket (configured via the bucket field in target properties).

Routing priority:

Pipeline bucket field (highest priority)
Configured buckets in buckets array (if bucket name matches)
Default bucket field (catch-all, lowest priority)

This multi-level routing enables flexible data distribution strategies, such as routing different log types to different buckets based on content analysis, source system, severity level, or any other runtime decision.

Templates

The following template variables can be used in file names:

Variable	Description	Example
`{{.Year}}`	Current year	`2024`
`{{.Month}}`	Current month	`01`
`{{.Day}}`	Current day	`15`
`{{.Timestamp}}`	Current timestamp in nanoseconds	`1703688533123456789`
`{{.Format}}`	File format	`json`
`{{.Extension}}`	File extension	`json`
`{{.Compression}}`	Compression type	`zstd`
`{{.TargetName}}`	Target name	`my_logs`
`{{.TargetType}}`	Target type	`awss3`
`{{.Table}}`	Bucket name	`logs`

Multipart Upload

Large files automatically use S3 multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple S3 buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require a schema. The schema value can be a Library schema name, a built-in schema name, a path to a schema file, or an inline JSON definition. See Avro and Parquet for the JSON definition format.

Examples

Basic Configuration

The minimum configuration for a JSON S3 target:

targets:
  - name: basic_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "datastream-logs"

Pipeline-Based Routing

Dynamic bucket routing using pipeline processors to analyze log content and route to appropriate buckets:

targets:
  - name: smart_routing_s3
    type: awss3
    pipelines:
      - dynamic_routing
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      buckets:
        - bucket: "security-events"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
        - bucket: "application-events"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
        - bucket: "system-events"
          name: "system-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
      bucket: "other-events"
      name: "other-{{.Timestamp}}.json"
      format: "json"

pipelines:
  - name: dynamic_routing
    processors:
      - set:
          field: "_vmetric.bucket"
          value: "security-events"
          if: "ctx.event_type == 'security'"
      - set:
          field: "_vmetric.bucket"
          value: "application-events"
          if: "ctx.event_type == 'application'"
      - set:
          field: "_vmetric.bucket"
          value: "system-events"
          if: "ctx.event_type == 'system'"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Multiple Buckets with Different Formats

Configuration for distributing data across multiple S3 buckets with different formats:

targets:
  - name: multi_bucket_export
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-west-2"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_s3
    type: awss3
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_s3
    type: awss3
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Synopsis​

Schema​

Configuration​

AWS Credentials​

Connection​

Files​

Scheduling​

Debug Options​

Details​

Authentication Methods​

IAM Permissions​

File Formats​

Compression​

File Management​

Bucket Routing​

Templates​

Multipart Upload​

Multiple Buckets​

Schema Requirements​

Examples​

Basic Configuration​

Pipeline-Based Routing​

Multiple Buckets with Catch-All​

Multiple Buckets with Different Formats​

Parquet Format​

High Reliability​

With Field Normalization​

Debug Configuration​

Synopsis

Schema

Configuration

AWS Credentials

Connection

Files

Scheduling

Debug Options

Details

Authentication Methods

IAM Permissions

File Formats

Compression

File Management

Bucket Routing

Templates

Multipart Upload

Multiple Buckets

Schema Requirements

Examples

Basic Configuration

Pipeline-Based Routing

Multiple Buckets with Catch-All

Multiple Buckets with Different Formats

Parquet Format

High Reliability

With Field Normalization

Debug Configuration