Oracle Cloud Object Storage

Oracle Cloud Long Term Storage

Synopsis

Creates a target that writes log messages to Oracle Cloud Infrastructure (OCI) Object Storage with support for various file formats and authentication methods. The target handles large file uploads efficiently with configurable rotation based on size or event count. OCI Object Storage stores the files with server-side encryption and integrates with the rest of Oracle Cloud.

Schema

- name: <string>
  description: <string>
  type: oracles3
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `oracles3`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Oracle Cloud Object Storage Credentials

Field	Required	Default	Description
`key`	Y	-	Oracle Cloud access key ID for Customer Secret Keys
`secret`	Y	-	Oracle Cloud secret access key
`session`	N	-	STS temporary session token (for short-lived credentials)
`region`	Y	-	OCI region identifier (e.g., `us-ashburn-1`, `eu-frankfurt-1`, `ap-tokyo-1`)
`endpoint`	Y	-	OCI Object Storage endpoint (format: `https://<namespace>.compat.objectstorage.<region>.oraclecloud.com`)

Connection

Field	Required	Default	Description
`part_size`	N	`5`	Multipart upload part size in megabytes (minimum 5MB)
`timeout`	N	`30`	Connection timeout in seconds
`field_format`	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
`bucket`	N*	-	Default OCI bucket name (used if `buckets` not specified)
`buckets`	N*	-	Array of bucket configurations for file distribution
`buckets.bucket`	Y	-	OCI bucket name
`buckets.name`	Y	-	File name template
`buckets.format`	N	`"json"`	Output format: `json`, `multijson`, `avro`, `parquet`
`buckets.compression`	N	-	Compression algorithm. See the Compression section below
`buckets.extension`	N	Matches `format`	File extension override
`buckets.schema`	N*	-	Schema reference (required for Avro and Parquet formats)
`name`	N	`"vmetric.{{.Timestamp}}.{{.Extension}}"`	Default file name template when `buckets` not used
`format`	N	`"json"`	Default output format when `buckets` not used
`compression`	N	`zstd`	Default compression when `buckets` not used
`extension`	N	Matches `format`	Default file extension when `buckets` not used
`schema`	N	-	Default schema reference when `buckets` not used
`max_size`	N	`33554432`	Maximum file size in bytes before rotation (default 32MB)
`batch_size`	N	`100000`	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to OCI Object Storage and a new file is created. Setting max_size to 0 does not disable rotation — an explicit 0 is replaced by the 32 MB default (MustInt64), so there is no unlimited setting. Raise the value instead.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
`debug.status`	N	`false`	Enable debug logging
`debug.dont_send_logs`	N	`false`	Process logs but don't send to target (testing)

Details

The Oracle Cloud Infrastructure Object Storage target writes log files to OCI Object Storage buckets in any of the supported file formats. OCI Object Storage encrypts the objects automatically, and Oracle Database and the analytics services can read them from the bucket.

Authentication

Requires OCI Customer Secret Keys for S3 compatibility. Customer Secret Keys can be generated through the OCI Console under User Settings. Each user can have up to two active Customer Secret Keys at a time.

Endpoint Configuration

The endpoint URL follows the pattern https://<namespace>.compat.objectstorage.<region>.oraclecloud.com where <namespace> is your OCI Object Storage namespace (typically your tenancy name) and <region> is your chosen OCI region identifier.

Finding Your Namespace

Your Object Storage namespace can be found in the OCI Console under Tenancy Details or by using the OCI CLI command oci os ns get.

Available Regions

Oracle Cloud Infrastructure is available in numerous regions worldwide:

Region Code	Location
`us-ashburn-1`	US East (Ashburn)
`us-phoenix-1`	US West (Phoenix)
`us-sanjose-1`	US West (San Jose)
`ca-toronto-1`	Canada Southeast (Toronto)
`ca-montreal-1`	Canada Southeast (Montreal)
`eu-frankfurt-1`	Germany Central (Frankfurt)
`eu-zurich-1`	Switzerland North (Zurich)
`eu-amsterdam-1`	Netherlands Northwest (Amsterdam)
`uk-london-1`	UK South (London)
`ap-tokyo-1`	Japan East (Tokyo)
`ap-osaka-1`	Japan Central (Osaka)
`ap-seoul-1`	South Korea Central (Seoul)
`ap-mumbai-1`	India West (Mumbai)
`ap-hyderabad-1`	India South (Hyderabad)
`ap-sydney-1`	Australia East (Sydney)
`ap-melbourne-1`	Australia Southeast (Melbourne)
`sa-saopaulo-1`	Brazil East (Sao Paulo)
`me-jeddah-1`	Saudi Arabia West (Jeddah)
`me-dubai-1`	UAE East (Dubai)

File Formats

Format	Description
`json`	Each log entry is written as a separate JSON line (JSONL format)
`multijson`	All log entries are written as a single JSON array
`avro`	Apache Avro format with schema
`parquet`	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	`zstd`	`deflate`, `snappy`, `zstd`
Parquet	`zstd`	`gzip`, `snappy`, `zstd`, `brotli`, `lz4`

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

Bucket Routing

The target supports flexible bucket routing through pipeline configuration or explicit bucket settings:

Configuration-based routing: Define multiple buckets in the target configuration, each with its own format, compression, and schema settings. Logs are routed to specific buckets based on configuration.

Pipeline-based routing: Use the bucket field in pipeline processors to dynamically route logs to different buckets at runtime. This enables conditional routing based on log content, source, or other attributes.

Catch-all routing: When a log doesn't match any specific bucket configuration or when no bucket field is set in the pipeline, logs are routed to the catch-all bucket (configured via the bucket field in target properties).

Routing priority:

Pipeline bucket field (highest priority)
Configured buckets in buckets array (if bucket name matches)
Default bucket field (catch-all, lowest priority)

This multi-level routing enables flexible data distribution strategies, such as routing different log types to different buckets based on content analysis, source system, severity level, or any other runtime decision.

Templates

The following template variables can be used in file names:

Variable	Description	Example
`{{.Year}}`	Current year	`2024`
`{{.Month}}`	Current month	`01`
`{{.Day}}`	Current day	`15`
`{{.Timestamp}}`	Current timestamp in nanoseconds	`1703688533123456789`
`{{.Format}}`	File format	`json`
`{{.Extension}}`	File extension	`json`
`{{.Compression}}`	Compression type	`zstd`
`{{.TargetName}}`	Target name	`my_logs`
`{{.TargetType}}`	Target type	`oracles3`
`{{.Table}}`	Bucket name	`logs`
`{{.Thread}}`	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	`3`
`{{.ServiceRoot}}`	The service root directory	`/opt/vmetric`

Multipart Upload

Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple OCI buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require a schema. The schema value can be a Library schema name, a built-in schema name, or an inline JSON definition. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup, so an Avro schema must be a name or inline JSON. See Avro and Parquet for the JSON definition format.

Storage Tiers

OCI Object Storage supports multiple storage tiers including Standard, Infrequent Access, and Archive for cost optimization based on access patterns and retention requirements.

Integration with Oracle Services

Oracle Autonomous Database, Oracle Analytics Cloud, and other Oracle Cloud services can read the objects directly from the bucket.

Examples

Basic Configuration

The minimum configuration for a JSON OCI Object Storage target:

targets:
  - name: basic_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "us-ashburn-1"
      endpoint: "https://mytenancy.compat.objectstorage.us-ashburn-1.oraclecloud.com"
      bucket: "datastream-logs"

Multiple Buckets

Configuration for distributing data across multiple OCI buckets with different formats:

targets:
  - name: multi_bucket_export
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "eu-frankfurt-1"
      endpoint: "https://mytenancy.compat.objectstorage.eu-frankfurt-1.oraclecloud.com"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "us-ashburn-1"
      endpoint: "https://mytenancy.compat.objectstorage.us-ashburn-1.oraclecloud.com"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "ap-tokyo-1"
      endpoint: "https://mytenancy.compat.objectstorage.ap-tokyo-1.oraclecloud.com"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_oci
    type: oracles3
    pipelines:
      - checkpoint
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "uk-london-1"
      endpoint: "https://mytenancy.compat.objectstorage.uk-london-1.oraclecloud.com"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "ap-sydney-1"
      endpoint: "https://mytenancy.compat.objectstorage.ap-sydney-1.oraclecloud.com"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "us-phoenix-1"
      endpoint: "https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Troubleshooting

The oracles3 target behaves the same way as the Amazon S3 target and reports the same errors for the same causes: refused credentials, a bucket that does not exist, region and endpoint mistakes, upload timeouts, throttling, and buckets that stay empty. Use the Troubleshooting section of Amazon S3 for the full list of errors, causes, and fixes. See Target Delivery Errors for how Director logs and retries target failures.

Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples there.

What differs for OCI Object Storage

Startup probes the account, not one bucket. Before anything is sent, the target makes one list-buckets call, so the credentials need an account-wide list permission. The user that owns the Customer Secret Key has to be allowed to list the buckets of the tenancy. A user whose policy only lets it write into one bucket cannot start the target, and you get api error AccessDenied after ListBuckets with nothing sent.
key and secret must both be set. They are used only when both resolve to a non-empty value. There is no instance-role or ambient credential fallback that can work here, so an empty value leaves the target with no credentials at all and it reports no EC2 IMDS role found instead. A ${VAR} or $secret{...} reference that resolves to an empty string fails the same way while the configuration still looks complete.
region and endpoint must both be set, and they must agree. Use the region identifier, such as us-ashburn-1, not the location name shown in the console. Set endpoint to the full URL, https://<namespace>.compat.objectstorage.<region>.oraclecloud.com, with the scheme included. Without endpoint the request goes to Amazon S3. Requests are signed with region, so an endpoint that serves another region typically answers with SignatureDoesNotMatch even though both values look correct on their own.
Name resolution and certificates cause most connection failures. Read the entry on the primary page that covers no such host, connection refused, and certificate signed by unknown authority. This target reads no TLS options, so a certificate issued by a private certificate authority has to be installed in the Director host's own trust store, and there is no setting that skips verification.

The endpoint has to carry your own namespace

The first label of the endpoint host is the Object Storage namespace of your tenancy, not a name you choose. A wrong value there typically leaves the hostname unresolvable, so the target fails at startup with no such host rather than with a credential error. Read the namespace from the OCI Console under Tenancy Details, or with oci os ns get, and paste it into endpoint.

note

No bucket name is checked at startup. The probe only confirms that the credentials can list the account, so a misspelled name under bucket or buckets passes startup and fails on the first upload routed to it.

tip

If a bucket holds only the last batch, check that every name template contains {{.Timestamp}}. A name built only from {{.Year}}, {{.Month}}, and {{.Day}} produces the same object key on every flush, and each upload replaces the one before it.

Synopsis​

Schema​

Configuration​

Oracle Cloud Object Storage Credentials​

Connection​

Files​

Scheduling​

Debug Options​

Details​

Authentication​

Endpoint Configuration​

Finding Your Namespace​

Available Regions​

File Formats​

Compression​

File Management​

Bucket Routing​

Templates​

Multipart Upload​

Multiple Buckets​

Schema Requirements​

Storage Tiers​

Integration with Oracle Services​

Examples​

Basic Configuration​

Multiple Buckets​

Multiple Buckets with Catch-All​

Parquet Format​

High Reliability​

With Field Normalization​

Debug Configuration​

Troubleshooting​

What differs for OCI Object Storage​

The endpoint has to carry your own namespace​

Synopsis

Schema

Configuration

Oracle Cloud Object Storage Credentials

Connection

Files

Scheduling

Debug Options

Details

Authentication

Endpoint Configuration

Finding Your Namespace

Available Regions

File Formats

Compression

File Management

Bucket Routing

Templates

Multipart Upload

Multiple Buckets

Schema Requirements

Storage Tiers

Integration with Oracle Services

Examples

Basic Configuration

Multiple Buckets

Multiple Buckets with Catch-All

Parquet Format

High Reliability

With Field Normalization

Debug Configuration

Troubleshooting

What differs for OCI Object Storage

The endpoint has to carry your own namespace