Google Cloud Storage

Google Cloud Long-Term Storage

Synopsis

Creates a target that writes log messages to Google Cloud Storage buckets with support for various file formats, authentication methods, and multipart uploads. The target handles large file uploads efficiently with configurable rotation based on size or event count.

Schema

- name: <string>
  description: <string>
  type: gcpstorage
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    credentials: <string>
    project_id: <string>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y	-	Must be gcpstorage
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Google Cloud Storage Credentials

Field	Required	Default	Description
credentials	N	-	Service account credentials JSON. Uses Application Default Credentials (ADC) if not provided
project_id	N	-	Google Cloud project ID. Accepted for readability. The bucket determines the project that is written to, so this value does not affect authentication or uploads

Connection

Field	Required	Default	Description
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
bucket	N*	-	Default GCS bucket name (acts as catch-all when buckets is also specified)
buckets	N*	-	Array of bucket configurations for file distribution
buckets.bucket	Y	-	GCS bucket name
buckets.name	Y	-	File name template
buckets.format	N	"json"	Output format: json, multijson, avro, parquet
buckets.compression	N	-	Compression algorithm (e.g., zstd, gzip, lz4, snappy). See the File Formats include below
buckets.extension	N	Matches format	File extension override
buckets.schema	N**	-	Schema reference (required for Avro and Parquet formats)
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	Default file name template (used with bucket for catch-all)
format	N	"json"	Default output format (used with bucket for catch-all)
compression	N	zstd	Default compression (used with bucket for catch-all)
extension	N	Matches format	Default file extension (used with bucket for catch-all)
schema	N	-	Default schema reference (used with bucket for catch-all)
max_size	N	33554432	Maximum file size in bytes before rotation (32 MB). An explicit 0 is replaced by that default, so it does not mean unlimited
batch_size	N	100000	Maximum number of messages per file

* = Either bucket or buckets must be specified.

** = Conditionally required for Avro and Parquet formats when using buckets.

note

When max_size is reached, the current file is uploaded to GCS and a new file is created. Setting max_size to 0 does not disable rotation — an explicit 0 is replaced by the 32 MB default (MustInt64), so there is no unlimited setting. Raise the value instead.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Google Cloud Storage target writes batched log records to one or more buckets, in any of the file formats listed below. GCS offers high durability (99.999999999%), strong consistency for read-after-write operations, and integration with Google Cloud's security and analytics ecosystem.

Authentication Methods

Supports service account credentials JSON provided via the credentials field. When deployed on Google Cloud infrastructure, can leverage Application Default Credentials without explicit credentials by omitting the credentials field.

IAM Permissions

The service account requires the following IAM role:

IAM Role	Role ID	Purpose
Storage Object Creator	roles/storage.objectCreator	Upload (create) objects in GCS buckets

Minimum permission: storage.objects.create. Add storage.objects.delete when an object name can repeat, because writing to a name that already exists replaces the object.

Storage Classes

Google Cloud Storage supports multiple storage classes for cost optimization:

Storage Class	Use Case
Standard	Frequently accessed data
Nearline	Data accessed less than once per month
Coldline	Data accessed less than once per quarter
Archive	Data accessed less than once per year

Available Regions

Google Cloud Storage is available in multiple regions worldwide:

Region Code	Location
us-central1	Iowa, USA
us-east1	South Carolina, USA
us-west1	Oregon, USA
europe-west1	Belgium
europe-west2	London, UK
europe-west3	Frankfurt, Germany
asia-east1	Taiwan
asia-northeast1	Tokyo, Japan
asia-southeast1	Singapore
australia-southeast1	Sydney, Australia

Bucket Routing

The target supports flexible bucket routing through pipeline configuration or explicit bucket settings:

Configuration-based routing: Define multiple buckets in the target configuration, each with its own format, compression, and schema settings. Logs are routed to specific buckets based on configuration.

Pipeline-based routing: Use the bucket field in pipeline processors to dynamically route logs to different buckets at runtime. This enables conditional routing based on log content, source, or other attributes.

Catch-all routing: When a log doesn't match any specific bucket configuration or when no bucket field is set in the pipeline, logs are routed to the catch-all bucket (configured via the bucket field in target properties).

Routing priority:

1. Pipeline bucket field (highest priority)
2. Configured buckets in buckets array (if bucket name matches)
3. Default bucket field (catch-all, lowest priority)

This multi-level routing enables flexible data distribution strategies, such as routing different log types to different buckets based on content analysis, source system, severity level, or any other runtime decision.

File Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	zstd	deflate, snappy, zstd
Parquet	zstd	gzip, snappy, zstd, brotli, lz4

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

note

The default compression is zstd, and it is not applied to the line formats json, jsonl, multijson and raw. Only gzip applies to those, and any other value leaves the object uncompressed. A default configuration therefore writes uncompressed JSON.

Templates

The following template variables can be used in file names:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	gcpstorage
{{.Table}}	Bucket name	logs
{{.Thread}}	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	3
{{.ServiceRoot}}	The service root directory	/opt/vmetric

Multiple Buckets

Single target can write to multiple GCS buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require a schema. The schema value can be a Library schema name, a built-in schema name, or an inline JSON definition. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup, so an Avro schema must be a name or inline JSON. See Avro and Parquet for the JSON definition format.

Integration with Google Cloud

GCS integrates with other Google Cloud services including BigQuery for analytics, Cloud Functions for serverless processing, and Cloud Logging for centralized logging.

Examples

Basic Configuration

The minimum configuration for a JSON GCS target:

targets:
  - name: basic_gcs
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      bucket: "datastream-logs"

Service Account Authentication

Configuration with explicit service account credentials:

targets:
  - name: gcs_service_account
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: |
        {
          "type": "service_account",
          "project_id": "my-project-123456",
          "private_key_id": "key-id",
          "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
          "client_email": "storage@my-project-123456.iam.gserviceaccount.com",
          "client_id": "123456789",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token"
        }
      bucket: "datastream-logs"

Pipeline-Based Routing

Dynamic bucket routing using pipeline processors to analyze log content and route to appropriate buckets:

targets:
  - name: smart_routing_gcs
    type: gcpstorage
    pipelines:
      - dynamic_routing
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      buckets:
        - bucket: "security-events"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-events"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "system-events"
          name: "system-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "other-events"
      name: "other-{{.Timestamp}}.json"
      format: "json"

pipelines:
  - name: dynamic_routing
    processors:
      - set:
          field: "_vmetric.bucket"
          value: "security-events"
          if: "ctx.event_type == 'security'"
      - set:
          field: "_vmetric.bucket"
          value: "application-events"
          if: "ctx.event_type == 'application'"
      - set:
          field: "_vmetric.bucket"
          value: "system-events"
          if: "ctx.event_type == 'system'"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Multiple Buckets with Different Formats

Configuration for distributing data across multiple GCS buckets with different formats:

targets:
  - name: multi_bucket_export
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_gcs
    type: gcpstorage
    pipelines:
      - checkpoint
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_gcs
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

BigQuery Integration

Configuration optimized for BigQuery data lake:

targets:
  - name: bigquery_ready
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      bucket: "bigquery-staging"
      name: "bq-import/{{.Year}}/{{.Month}}/{{.Day}}/data-{{.Timestamp}}.json"
      format: "json"
      compression: "gzip"
      max_size: 1073741824

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_gcs
    type: gcpstorage
    properties:
      project_id: "my-project-123456"
      credentials: "${GCP_CREDENTIALS_JSON}"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Troubleshooting

This section covers the errors you are most likely to see with the gcpstorage target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission is missing?

The target never creates a bucket and never inspects one, so the permission set is small. Match the error you see against this table.

Error text	Missing permission or role	Must be granted on
googleapi: Error 403: ..., forbidden on the first upload, with storage.objects.create in the message	storage.objects.create, in roles/storage.objectCreator	Each bucket named in bucket and in buckets
googleapi: Error 403: ..., forbidden that starts only on the second flush, with storage.objects.delete in the message	storage.objects.delete, in roles/storage.objectUser or roles/storage.objectAdmin	The bucket whose object names repeat
googleapi: Error 404: ..., notFound	No permission is missing. The bucket has to exist before you send to it	Create the bucket yourself, in a project the service account can see

The text that replaces ... between Error 403: and forbidden comes from Google, not from Director, so the exact wording can change. It typically names the service account and the permission that was refused.

You need storage.objects.delete only when your object names repeat. Replacing an object that already exists is treated as a delete, so an identity holding roles/storage.objectCreator alone typically writes the first object and is refused from then on. Adding {{.Timestamp}} to name is the better fix, because every flush then writes a new object and the creator role is enough on its own. See the entry on repeated object names below.

"failed to create GCS client" at startup

[Error] [director] [target-<target id>] [gcs_service_account] Failed to reinitialize target "gcs_service_account" (attempt 3). Reason: failed to create GCS client: dialing: invalid character '$' looking for beginning of value

Cause: credentials could not be read as a service account key. The text after dialing: tells you which way it failed.

Reason text	What it means
invalid character '$' looking for beginning of value	credentials holds an environment variable reference such as ${GCP_CREDENTIALS_JSON} that was never expanded, so the literal placeholder arrived where the key was expected. The variable is not set for the Director service
unexpected end of JSON input, or another character reported as invalid	The key file is truncated or mangled. A frequent cause is a private key whose line breaks were not written as \n inside the JSON string
credentials: unsupported unidentified file type	The JSON parsed but carries no type field. Only the inner part of a key file was pasted, or the file is not a Google credential at all
credentials: unsupported filetype "..."	The type field names a credential kind the client does not accept. Use a service account key
auth: email must be provided or auth: private key must be provided	client_email or private_key is missing from the key file
credentials: could not find default credentials.	credentials is empty and no Application Default Credentials were found. The host is not a Google Cloud instance, GOOGLE_APPLICATION_CREDENTIALS is not set for the service user, and there is no gcloud credential file in its home directory

Fix: paste the complete service account key into credentials, or set the environment variable it refers to and restart the service. Export the variable for the Director service itself, not only in the shell you tested from. To use Application Default Credentials instead, leave credentials empty and either run Director on a Google Cloud instance with a service account attached, or point GOOGLE_APPLICATION_CREDENTIALS at a key file the service user can read.

Nothing is sent while this lasts. Incoming data waits in the Director queue and is delivered once the target starts, so no restart is needed once you fix the cause.

"googleapi: Error 403" when an object is created

[Error] [director] [target-<target id>] [gcs_service_account] Sender worker 2 execute() failed for gcs_service_account: target broken: failed to finalize target cache: googleapi: Error 403: ..., forbidden

Cause: the target started and buffered records, but Google refused the upload. Three different problems produce this one message. The identity has no storage.objects.create on that bucket. Or the object name already exists and the identity may not replace it. Or the bucket sits in a project the identity cannot see, which Google answers with 403 rather than 404.

Fix: grant roles/storage.objectCreator on the bucket to the service account in credentials. If the message names storage.objects.delete, your object names repeat, so add {{.Timestamp}} to name rather than granting delete. Then confirm the bucket is in a project that service account can see.

Nothing is lost. The batch is discarded from memory, the payloads stay queued, and they are redelivered until the upload is accepted.

note

A green connection status does not prove the permissions are right. Director writes nothing to your bucket until the first flush, so a missing role first shows up here rather than at startup.

"googleapi: Error 404" and a bucket that cannot be found

[Error] [director] [target-<target id>] [reliable_gcs] Sender worker 0 Finalize failed on flush for target "reliable_gcs": googleapi: Error 404: ..., notFound

Cause: there is no such bucket. The message typically says that the specified bucket does not exist. The target never checks a bucket at startup and never creates one, so a typo passes startup and fails at the first upload. Names under buckets fail the same way, and only once a record is routed to them.

Fix: create the bucket, or correct bucket and every buckets[].bucket entry. A bucket that does exist but belongs to a project the service account cannot see is refused with 403 instead, so check the project before you assume the name is wrong. Nothing is lost while this lasts, and the batch is redelivered until it is accepted.

"context deadline exceeded" while uploading

[Error] [director] [target-<target id>] [gcs_archive] Sender worker 3 execute() failed for gcs_archive: target broken: failed to finalize target cache: context deadline exceeded

Cause: timeout bounds the whole upload of one object, and it defaults to 30 seconds. With the 32 MB max_size default that asks for roughly 1.1 MB/s sustained, which a slow, distant, or proxied link does not always deliver. Every retry starts again from the first byte, so it fails the same way each time.

Fix: raise timeout, lower max_size so each object is smaller, or do both. Nothing is lost and the batch is redelivered, but the target makes no progress at all until one of those values changes.

warning

timeout: 0 does not mean no limit. It is read as a deadline of zero seconds, and every upload then fails instantly with this message. A value that is not a number behaves the same way. Remove the line to get the 30 second default back.

"no such host", "connection refused", or "certificate signed by unknown authority"

[Error] [director] [target-<target id>] [reliable_gcs] Sender worker 1 execute() failed for reliable_gcs: target broken: failed to finalize target cache: Post "https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?...": dial tcp: lookup storage.googleapis.com on 10.0.0.2:53: no such host

Cause: Director could not reach Google. The text after the upload URL says which step failed.

• no such host means the Director host cannot resolve storage.googleapis.com. The address after on is the resolver that answered.
• connection refused, i/o timeout, or net/http: TLS handshake timeout means outbound 443 is blocked or is being dropped on the way out.
• proxyconnect tcp: ... means the proxy named in HTTPS_PROXY could not be reached.
• x509: certificate signed by unknown authority means something is presenting a certificate from a private certificate authority, which is usually a TLS-inspecting proxy.

Fix: allow outbound HTTPS from the Director host to storage.googleapis.com for uploads and to oauth2.googleapis.com for tokens. Set HTTPS_PROXY and NO_PROXY on the Director service when a proxy is required. For a private certificate authority, install its certificate in the Director host's own trust store. This target has no option for a custom authority and none for skipping verification.

Nothing is lost while this lasts, and the batch is redelivered until it succeeds.

"invalid_grant", or a private key that cannot be parsed

[Error] [director] [target-<target id>] [gcs_service_account] Sender worker 1 execute() failed for gcs_service_account: target broken: failed to finalize target cache: Post "https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?...": auth: "invalid_grant" "..."

Cause: the key file is well formed, so the target started, but Google would not exchange it for a token. The access token is fetched with the first upload and not at startup, which is why a key problem can surface long after the connection status went green. An invalid_grant reason typically means the key was deleted or rotated, the service account is disabled, or the Director host clock has drifted. A reason of failed to parse private key. Tried PKCS8, PKCS1, and EC formats. or private key is not a signer means the private_key value itself is mangled, usually by broken line breaks or a missing end marker.

Fix: issue a new key for the service account and paste the complete file into credentials, keeping the \n escapes inside the private_key string intact. Check that the service account is still enabled, and that the clock on the Director host is synchronised. Nothing is lost, and the batch is redelivered until a token is issued.

"googleapi: Error 429", or a 5xx error

Cause: Google is throttling the requests, or the service is failing them. Small objects make this more likely, because a new object is written every time batch_size or max_size is reached, and each writer thread rotates on its own.

Fix: this usually clears by itself, since the batch is redelivered until it is accepted. If it persists, raise batch_size and max_size so each object carries more, or set interval or cron so the target flushes on a schedule with a single writer, as described under Scheduling and Pool Behavior. Spreading the keys over more prefixes, by putting {{.Year}}/{{.Month}}/{{.Day}} at the front of name, also helps.

"file holder not found"

[Error] [director] [target-<target id>] [smart_routing_gcs] Sender worker 0 execute() failed for smart_routing_gcs: target broken: file holder not found

Cause: the target has buckets but no top-level bucket, and a record arrived whose _vmetric.bucket is empty or names a bucket that is not in the list. The comparison is case-sensitive, so Security-Events does not match security-events. There is nowhere to write the record.

Fix: add a top-level bucket as the catch-all, or make the pipeline set _vmetric.bucket on every record to a value that matches a buckets[].bucket entry exactly. Nothing is lost, but the payload is redelivered every few seconds and fails the same way each time, which holds up everything queued behind it, so treat this one as urgent.

Configuration mistakes reported as reinitialize failures

The bucket list is checked when the target starts, not when you save it, so these reach you through the same reinitialize loop as a connection failure. The attempt counter climbs for as long as the problem lasts and nothing is sent. Correct the reason and the next attempt picks it up, with no restart.

Reason text	Fix
no valid buckets found	Set bucket, or at least one entry under buckets
invalid bucket configuration: bucket is required	Every entry under buckets needs a bucket
invalid bucket configuration: name is required	Every entry under buckets needs its own name. Only the top-level name has a default, and only when you leave the key out altogether
invalid bucket configuration: schema is required for parquet format	Add schema wherever format is avro or parquet, including entries that inherit the format from the top level
failed to parse Avro schema: ..., failed to parse Parquet schema: ..., invalid schema format: ...	The schema text is malformed, or a built-in schema name is misspelled and was read as an inline field:type list
schema file not found for: ..., schema directory does not exist: ...	The schema is not deployed on the Director host under the name you referenced
failed to create Avro encoder: ...	The schema parsed, but the encoder rejects a type in it. Simplify that type

A record that does not fit the schema fails later, during delivery, carrying the encoder's own text such as avro: ... is unsupported for Avro string. That record is redelivered rather than dropped, so it blocks everything queued behind it until you correct the schema or normalize the field in a pipeline.

Only the last batch is in the bucket

Nothing fails here. Every upload succeeds, the counters move, and the connection status stays healthy.

Cause: the object name expands to the same value on every flush. A name such as logs-{{.Year}}-{{.Month}}-{{.Day}}.json is the same string for a whole day, so each flush replaces the object the flush before it wrote, and only the last batch of the day survives. The target flushes far more often than once a day. It rotates whenever batch_size or max_size is reached, and again on every scheduled flush.

How to notice it: list the bucket after a busy hour. One object per bucket per day, whose size and update time keep changing, is the symptom. So is an event count in Director far higher than the number of records you can find in the bucket.

Fix: put {{.Timestamp}} in name, and in the name of every entry under buckets. The timestamp is in nanoseconds, so each flush lands as its own object:

name: "logs-{{.Year}}-{{.Month}}-{{.Day}}-{{.Timestamp}}.json"

Keep the date parts if you use them to organise the bucket. They group the objects, they do not make the name unique.

Three variations produce the same collapse:

• A broken template, such as a missing closing brace, leaves the text unexpanded in the object name, and an unknown variable renders as <no value>. Either way the name stops varying. Use only the variables listed under Templates above, and check the object names after the first flush.
• A name that renders as invalid UTF-8 is rejected outright, with storage: object name ... is not valid UTF-8 instead of a silent overwrite.
• In immediate mode each writer thread keeps its own object, so you may find one object per thread rather than one in total. That is the same problem, divided by the number of threads.

tip

Turning on object versioning for the bucket preserves the overwritten objects, which is worth doing while you investigate. It is not a substitute for a unique name.

warning

With roles/storage.objectCreator alone, a repeated object name typically fails with 403 from the second flush onwards, because replacing an object needs storage.objects.delete. The same mistake therefore appears either as silent data loss or as a permission error, depending on which roles you granted.

The target is healthy but the objects are missing or wrong

Check these in order.

1. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No object is written, no counter moves, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well. Remove the flag when you have finished testing.

   Log sending is disabled for this target (debug_gcs). Logs will be processed by the pipeline but will not be sent to the target.

2. The records landed in the catch-all bucket. When _vmetric.bucket names a bucket that is not under buckets and a top-level bucket exists, the record is written to the catch-all without a warning. Look there before you assume the data is gone. Once two or more buckets have received data in the same cycle, the Locations: suffix on the Completed processing of ... target logs line shows how many events went where.

3. The objects are not compressed. For the line formats only gzip is applied. Any other value, including the default of zstd, uploads uncompressed while {{.Compression}} in the name still claims otherwise. Set compression: gzip for json and multijson. Avro accepts deflate, snappy, and zstd, and Parquet accepts gzip, snappy, zstd, brotli, and lz4. An unrecognised codec is ignored rather than reported, and the object is written uncompressed.

4. The objects are not in the format their name claims. A format value that is not recognised falls back to JSON lines while extension keeps the value you typed, so the objects are named for a format they do not contain. Use json, multijson, avro, or parquet, in lower case.

5. Fields are missing from the records. With avro and parquet every record is reshaped to schema before it is written, so a field that is not in the schema never reaches the bucket. Extend the schema to keep it.

6. Objects overshoot max_size. The limit is measured on the buffer after encoding and compression, and Parquet holds rows back until a row group is complete, so a Parquet object can grow past the limit by up to one row group before it is written out.