MinIO

MinIO Long Term Storage

Synopsis

Creates a target that writes log messages to MinIO object storage with support for various file formats and authentication methods. The target handles large file uploads efficiently with configurable rotation based on size or event count. MinIO provides high-performance, Kubernetes-native object storage that can be deployed on-premises or in the cloud.

Schema

- name: <string>
  description: <string>
  type: minio
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be minio
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

MinIO Credentials

Field	Required	Default	Description
key	Y	-	MinIO access key
secret	Y	-	MinIO secret key
session	N	-	STS temporary session token (for short-lived credentials)
region	N	-	MinIO region. Left unset, no region is passed and the AWS SDK resolves one from the environment; most MinIO deployments accept us-east-1
endpoint	Y	-	MinIO server endpoint URL (e.g., http://minio-server:9000 or https://minio.example.com)

Connection

Field	Required	Default	Description
part_size	N	5	Multipart upload part size in megabytes (minimum 5MB)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
bucket	N*	-	Default MinIO bucket name (used if buckets not specified)
buckets	N*	-	Array of bucket configurations for file distribution
buckets.bucket	Y	-	MinIO bucket name
buckets.name	Y	-	File name template
buckets.format	N	"json"	Output format: json, multijson, avro, parquet
buckets.compression	N	-	Compression algorithm. See the Compression section below
buckets.extension	N	Matches format	File extension override
buckets.schema	N*	-	Schema reference (required for Avro and Parquet formats)
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	Default file name template when buckets not used
format	N	"json"	Default output format when buckets not used
compression	N	zstd	Default compression when buckets not used
extension	N	Matches format	Default file extension when buckets not used
schema	N	-	Default schema reference when buckets not used
max_size	N	33554432	Maximum file size in bytes before rotation (default 32MB)
batch_size	N	100000	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to MinIO and a new file is created. Setting max_size to 0 does not disable rotation — an explicit 0 is replaced by the 32 MB default (MustInt64), so there is no unlimited setting. Raise the value instead.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The MinIO target provides high-performance object storage integration with comprehensive file format support. MinIO is designed for private cloud infrastructure and offers enterprise features like versioning, lifecycle management, and replication.

Authentication

Requires MinIO access credentials. Access keys can be created through the MinIO Console or mc (MinIO Client) command-line tool. Keys can be scoped with specific permissions and bucket access policies.

Endpoint Configuration

The endpoint URL points to your MinIO server deployment. This can be a local deployment (e.g., http://localhost:9000), an internal network address (e.g., http://minio-service:9000), or a public domain with TLS (e.g., https://minio.example.com).

Deployment Flexibility

MinIO can be deployed in various environments including on-premises data centers, private clouds, public clouds, edge locations, and Kubernetes clusters. The target works with any of them.

File Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	zstd	deflate, snappy, zstd
Parquet	zstd	gzip, snappy, zstd, brotli, lz4

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

Bucket Routing

The target supports flexible bucket routing through pipeline configuration or explicit bucket settings:

Configuration-based routing: Define multiple buckets in the target configuration, each with its own format, compression, and schema settings. Logs are routed to specific buckets based on configuration.

Pipeline-based routing: Use the bucket field in pipeline processors to dynamically route logs to different buckets at runtime. This enables conditional routing based on log content, source, or other attributes.

Catch-all routing: When a log doesn't match any specific bucket configuration or when no bucket field is set in the pipeline, logs are routed to the catch-all bucket (configured via the bucket field in target properties).

Routing priority:

1. Pipeline bucket field (highest priority)
2. Configured buckets in buckets array (if bucket name matches)
3. Default bucket field (catch-all, lowest priority)

This multi-level routing enables flexible data distribution strategies, such as routing different log types to different buckets based on content analysis, source system, severity level, or any other runtime decision.

Templates

The following template variables can be used in file names:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	minio
{{.Table}}	Bucket name	logs
{{.Thread}}	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	3
{{.ServiceRoot}}	The service root directory	/opt/vmetric

Multipart Upload

Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple MinIO buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require a schema. The schema value can be a Library schema name, a built-in schema name, or an inline JSON definition. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup, so an Avro schema must be a name or inline JSON. See Avro and Parquet for the JSON definition format.

Performance Optimization

MinIO is optimized for high-throughput workloads and can handle concurrent uploads efficiently. Consider using erasure coding and distributed mode for enhanced performance and reliability.

Examples

Basic Configuration

The minimum configuration for a JSON MinIO target:

targets:
  - name: basic_minio
    type: minio
    properties:
      key: "minioadmin"
      secret: "minioadmin"
      endpoint: "http://minio-server:9000"
      bucket: "datastream-logs"

Multiple Buckets

Configuration for distributing data across multiple MinIO buckets with different formats:

targets:
  - name: multi_bucket_export
    type: minio
    properties:
      key: "analytics-user"
      secret: "analytics-password-secure"
      endpoint: "https://minio.example.com"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: minio
    properties:
      key: "minio-user"
      secret: "minio-password-secure"
      endpoint: "https://minio.example.com"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: minio
    properties:
      key: "analytics-user"
      secret: "analytics-password-secure"
      endpoint: "http://10.0.1.100:9000"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_minio
    type: minio
    pipelines:
      - checkpoint
    properties:
      key: "backup-user"
      secret: "backup-password-secure"
      endpoint: "https://minio-backup.example.com"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_minio
    type: minio
    properties:
      key: "normalized-user"
      secret: "normalized-password-secure"
      endpoint: "http://minio.internal:9000"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_minio
    type: minio
    properties:
      key: "test-user"
      secret: "test-password"
      endpoint: "http://localhost:9000"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Kubernetes Deployment

Configuration for MinIO deployed in Kubernetes:

targets:
  - name: k8s_minio
    type: minio
    properties:
      key: "k8s-minio-user"
      secret: "k8s-minio-password-secure"
      endpoint: "http://minio.default.svc.cluster.local:9000"
      bucket: "application-logs"
      name: "logs/{{.Year}}/{{.Month}}/{{.Day}}/{{.Timestamp}}.json"
      format: "json"
      compression: "gzip"

Troubleshooting

The minio target behaves the same way as the Amazon S3 target and reports the same errors for the same causes: refused credentials, a bucket that does not exist, region and endpoint mistakes, upload timeouts, throttling, and buckets that stay empty. Use the Troubleshooting section of Amazon S3 for the full list of errors, causes, and fixes. See Target Delivery Errors for how Director logs and retries target failures.

Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples there.

What differs for MinIO

• Startup probes the account, not one bucket. Before anything is sent, the target makes one list-buckets call, so the credentials need an account-wide list permission. A MinIO user whose policy covers a single bucket typically fails that probe, even though the same user can write objects into that bucket. You get api error AccessDenied after ListBuckets. Add s3:ListAllMyBuckets on * to the user's policy, and nothing is sent until you do.

• key and secret must both be set. They are used only when both resolve to a non-empty value. There is no instance-role or ambient credential fallback that can work here, so an empty value leaves the target with no credentials at all and it reports no EC2 IMDS role found instead. A ${VAR} or $secret{...} reference that resolves to an empty string fails the same way while the configuration still looks complete.

• region must hold a value, and endpoint must be the full URL. MinIO accepts any region value, commonly us-east-1, but the value is used to sign the request and cannot be empty. Left unset, the target reports A region must be set when sending requests to S3. Set endpoint to the full server URL with the scheme and the port, such as https://minio.example.com or http://minio-server:9000, and use http:// only inside a trusted network.

• A private certificate authority is the common failure here. A MinIO server presenting its own certificate fails the startup probe with certificate signed by unknown authority. This target reads no TLS options, so there is no setting to trust a custom authority or to skip verification: install the authority's certificate in the Director host's own trust store. Read the entry on the primary page that covers that message along with no such host and connection refused, and leave use_path_style unset so the custom endpoint keeps path-style addressing.

An internal endpoint sent through the proxy

If HTTPS_PROXY is set for the Director service, a request to a MinIO host on a routable internal address is sent through the proxy unless that host is listed in NO_PROXY. Loopback and link-local addresses always go direct, so a server that works from http://localhost:9000 can still fail from another host. You typically see connection refused, i/o timeout, or a proxyconnect tcp message. Add the MinIO host to NO_PROXY for the Director service itself, not only in your own shell, and restart the service so it reads the change.

note

No bucket name is checked at startup. The probe only confirms that the credentials can list the account, so a misspelled name under bucket or buckets passes startup and fails on the first upload routed to it.

tip

If a bucket holds only the last batch, check that every name template contains {{.Timestamp}}. A name built only from {{.Year}}, {{.Month}}, and {{.Day}} produces the same object key on every flush, and each upload replaces the one before it.

Nothing is lost while any of these lasts. Incoming data stays queued and the target retries until you fix the cause.