Google Cloud Logging

GCP Logging Target

Synopsis

The Google Cloud Logging target forwards events to Google Cloud Logging (formerly Stackdriver Logging) with configurable severity levels, labels, and authentication methods.

Schema

- name: <string>
  description: <string>
  type: gcplogging
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    log_name: <string>
    authentication:
      method: <auto|manual|secret>
    credentials: <string>
    severity: <string>
    labels: <map>
    batch_size: <integer>
    timeout: <integer>
    max_retries: <integer>
    retry_delay: <integer>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be gcplogging
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
project_id	Y	-	Google Cloud project ID. A full resource name such as folders/<id>, billingAccounts/<id> or organizations/<id> is also accepted, and writes to that parent instead
log_name	Y	-	Log name within the project

Authentication

Field	Required	Default	Description
authentication.method	N	auto	Authentication method: auto, manual, secret
credentials	N*	-	JSON service account credentials (inline string or environment variable expansion)

* = Conditionally required when authentication.method is manual or secret.

Log Settings

Field	Required	Default	Description
severity	N	DEFAULT	Default log severity level. See Severity Levels below
labels	N	-	Map of custom labels to attach to all log entries

Batch Configuration

Field	Required	Default	Description
batch_size	N	1000	Maximum log entries per batch
timeout	N	30	Request timeout in seconds
max_retries	N	0	Retry attempts for a failed send. Leave it at 0. At 1 or more a retry after a failed flush can report success without the records having been delivered
retry_delay	N	1	Delay between retries in seconds

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Severity Levels

Valid severity levels for Google Cloud Logging:

Severity	Description
DEFAULT	Default severity (no specific level)
DEBUG	Debug or trace information
INFO	Informational messages
NOTICE	Normal but significant events
WARNING	Warning events
ERROR	Error events
CRITICAL	Critical events requiring immediate action
ALERT	Alert requiring immediate notification
EMERGENCY	Emergency requiring immediate response

Authentication Methods

Auto (Default):

• Uses Application Default Credentials (ADC)
• Checks GOOGLE_APPLICATION_CREDENTIALS environment variable
• Falls back to compute metadata service for GCE/GKE

Manual:

• Inline JSON service account credentials
• Credentials embedded directly in configuration

Secret:

• Service account credentials from environment variable
• More secure than inline credentials for production

IAM Permissions

The service account requires the following IAM role:

IAM Role	Role ID	Purpose
Logs Writer	roles/logging.logWriter	Write log entries to Cloud Logging

Minimum permissions: logging.logEntries.create

Log Entry Structure

Each log entry sent to Google Cloud Logging includes:

• Timestamp: Event timestamp from pipeline
• Severity: Configured or default severity level
• Payload: Event message content
• Labels: Custom labels for filtering and organization

Labels for Log Organization

Labels enable efficient log filtering and organization:

• Resource labels: Identify the source resource
• User labels: Custom categorization
• System labels: Automatic GCP-assigned labels

Labels are key-value pairs attached to every log entry.

Performance Considerations

Batch Processing:

• Events are buffered until batch_size is reached
• Flush occurs on batch limit or service shutdown
• Larger batches reduce API calls but increase latency

Retry Logic:

• Failed sends are retried up to max_retries times
• Exponential backoff between retries using retry_delay
• Failed sends are redelivered from the queue rather than dropped

Examples

Basic Configuration

Sending logs to Google Cloud Logging using auto authentication from Application Default Credentials...

targets: - name: gcp-logs type: gcplogging properties: project_id: my-project-id log_name: application-logs authentication: method: auto

With Service Account

Using explicit service account credentials for authentication...

targets: - name: gcp-logs-manual type: gcplogging properties: project_id: my-project-id log_name: security-logs authentication: method: manual credentials: | { "type": "service_account", "project_id": "my-project-id", "private_key_id": "key-id", "private_key": "-----BEGIN PRIVATE KEY-----\n...", "client_email": "logging@my-project.iam.gserviceaccount.com", "client_id": "123456789", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token" }

With Secret Authentication

Loading service account credentials from environment variable for better security...

targets: - name: gcp-logs-secret type: gcplogging properties: project_id: my-project-id log_name: audit-logs authentication: method: secret credentials: "${GCP_LOGGING_CREDENTIALS}"

With Severity and Labels

Configuring specific severity level and custom labels for log organization...

targets: - name: gcp-logs-labeled type: gcplogging properties: project_id: my-project-id log_name: firewall-logs severity: WARNING labels: environment: production application: firewall datacenter: us-central1 authentication: method: auto

High-Volume Configuration

Optimizing for high-volume log ingestion with larger batches and retry configuration...

targets: - name: gcp-logs-high-volume type: gcplogging properties: project_id: my-project-id log_name: access-logs batch_size: 1000 timeout: 30 retry_delay: 1 authentication: method: secret credentials: "${GCP_LOGGING_CREDENTIALS}"

Error Severity

Forwarding error logs with ERROR severity for immediate visibility...

targets: - name: gcp-error-logs type: gcplogging properties: project_id: my-project-id log_name: application-errors severity: ERROR labels: log_type: error alert: true authentication: method: auto

Debug Logs

Sending debug-level logs for development and troubleshooting...

targets: - name: gcp-debug-logs type: gcplogging properties: project_id: my-project-id log_name: debug-logs severity: DEBUG labels: environment: development purpose: debugging authentication: method: auto

Production Configuration

Production-ready configuration with performance tuning, retry logic, and comprehensive labels...

targets: - name: gcp-logs-production type: gcplogging properties: project_id: production-project log_name: production-logs severity: INFO batch_size: 1000 timeout: 30 retry_delay: 2 labels: environment: production application: datastream region: us-central1 team: platform compliance: required authentication: method: secret credentials: "${GCP_LOGGING_CREDENTIALS}"

Troubleshooting

This section covers the errors you are most likely to see with the gcplogging target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

Every record fails with "cannot unmarshal string into Go value of type ..."

[Error] [director] [target-<target id>] [gcp-logs] Sender worker 2 execute() failed for gcp-logs: target broken: failed to finalize target cache: failed to send logs to Google Cloud Logging after 0 retries: saw 1000 errors; last: logging: json.Unmarshal: json: cannot unmarshal string into Go value of type map[string]interface {}

Symptom: the target starts without an error, and then every flush fails with this line. No entry reaches Cloud Logging. The number in saw N errors is the number of records in the batch, so it follows batch_size. It is the same for every record, whatever the message content, project_id, log_name, severity, labels, or authentication method.

This is not caused by your configuration, and it cannot be resolved by changing any setting on this page. Raise it with VirtualMetric support and attach the log line exactly as it appears.

Keep max_retries at its default of 0 in the meantime. That does not make delivery work, but at 1 or more the same failure can be counted as a delivery, as described under The target is healthy but nothing arrives, or the entries are not what you configured below.

Which permission is missing?

The connection to Cloud Logging is not opened while the target starts. Credentials are read at startup, but a missing role, a wrong project, and every network problem surface only at the first flush, so the target can report healthy for a while before any of the errors below appear. Match the error you see against this table.

Error text	Missing role or permission	Resource
rpc error: code = PermissionDenied on flush, typically with a description saying the caller does not have permission	roles/logging.logWriter, that is the logging.logEntries.create permission	The project named in project_id
rpc error: code = PermissionDenied on flush, typically with a description naming the API and saying it has not been used in the project or is disabled	The Cloud Logging API, logging.googleapis.com, enabled	The project named in project_id
rpc error: code = Unauthenticated on flush	A valid, unrevoked service-account key, and a host clock that is not skewed	The key in credentials, or the Application Default Credentials of the Director host
failed to create Google Cloud Logging client: credentials: ... at startup	A service-account key JSON whose type is service_account	The credentials value, when authentication.method is manual or secret
failed to create Google Cloud Logging client: credentials: could not find default credentials.	Application Default Credentials readable by the Director service user	The Director host, when authentication.method is auto

Nothing else has to be granted. The OAuth scope for writing log entries is requested for you, and the log named in log_name does not have to exist beforehand, because Cloud Logging creates it on the first write. The same role applies when project_id names a folder, a billing account, or an organization instead of a project. Those have to be written as a full resource name, such as organizations/123456, because a bare ID is read as a project ID, and the role is granted at that level.

"failed to create Google Cloud Logging client" with a credentials error

[Error] [director] [target-<target id>] [gcp-logs-secret] Failed to reinitialize target "gcp-logs-secret" (attempt 3). Reason: failed to create Google Cloud Logging client: invalid character '$' looking for beginning of value

Cause: the credentials could not be read, so the target never starts. The text after the wrapper says which form the problem takes.

Text after failed to create Google Cloud Logging client:	What it means
invalid character '$' looking for beginning of value	credentials is an environment variable reference such as "${GCP_LOGGING_CREDENTIALS}" and the variable is not set for the Director process. The reference is then passed on as the literal text you typed and read as key JSON. An unset reference does not count as empty, so it passes the required-field check first
invalid character at another character, or a message about the JSON ending early	The inline JSON is malformed. YAML quoting typically stripped the braces or the quotes, or the paste was truncated
credentials: unsupported unidentified file type	The JSON has no type field
credentials: unsupported filetype "..."	The type field is not service_account. An API key or an OAuth client JSON reads like this
credentials: could not parse key: failed to parse private key. Tried PKCS8, PKCS1, and EC formats.	The private_key value was altered. The \n escapes inside the JSON string were typically turned into real line breaks, or removed, when the key was pasted
credentials: could not find default credentials.	authentication.method is auto and the host has no Application Default Credentials
open ..., naming a file	GOOGLE_APPLICATION_CREDENTIALS points to a file that does not exist, or that the Director service user cannot read

Fix: with manual or secret, paste the key file exactly as you downloaded it and use a YAML block scalar so nothing is reinterpreted:

credentials: |
  {
    "type": "service_account",
    "project_id": "my-project",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
  }

For an environment variable reference, set the variable in the environment of the Director service, not only in the shell you tested from. With auto, set GOOGLE_APPLICATION_CREDENTIALS for the Director service user, or switch to secret and supply the key yourself. On a host that is not a Google Cloud instance, credential detection also probes the metadata service on each attempt, so a failing auto target takes a few seconds per attempt.

Nothing is sent while any of this lasts. Incoming data stays queued and is retried until you fix the cause, and no restart is needed once the value is correct.

"code = PermissionDenied" on flush

[Error] [director] [target-<target id>] [gcp-logs-production] Sender worker 4 Finalize failed on flush for target "gcp-logs-production": failed to send logs to Google Cloud Logging after 0 retries: saw 1 errors; last: rpc error: code = PermissionDenied desc = ...

Cause: the credentials were accepted and the write was refused. The description after desc = names the reason, and it is typically one of two: the service account does not hold roles/logging.logWriter on the project in project_id, or the Cloud Logging API is not enabled on that project. A service account that belongs to another project lands here too, unless it has been granted the role on the project you are writing to.

Fix: in the Google Cloud console, open IAM & Admin for the project in project_id and grant the service account the Logs Writer role, roles/logging.logWriter. Then confirm the Cloud Logging API is enabled on the same project. A role change is picked up on a later attempt, so no restart is needed.

No data is lost. The failed batch is discarded from memory, the payloads stay queued, and they are redelivered after the target restarts itself. Expect the queue to grow until the role is in place.

"code = NotFound", a project that does not exist, or an API that is not enabled

[Error] [director] [target-<target id>] [gcp-logs] Sender worker 1 Finalize failed on flush for target "gcp-logs": failed to send logs to Google Cloud Logging after 0 retries: saw 1 errors; last: rpc error: code = NotFound desc = ...

Cause: the project named in project_id cannot be resolved, or the Cloud Logging API has never been enabled on it. A project that does not exist is not always reported as NotFound. It is typically refused as PermissionDenied instead, so check this entry and the one above against the same symptom.

Fix: set project_id to the bare project ID, for example my-project. Do not paste a console URL or a resource path. projects/my-project is accepted, but any other prefix stops the target from starting with parent parameter must start with 'projects/' 'folders/' 'billingAccounts/' or 'organizations/'. Then enable the Cloud Logging API on that project, from APIs & Services in the console or with gcloud services enable logging.googleapis.com --project my-project.

No data is lost while this lasts. The payloads stay queued and are redelivered until the writes are accepted.

"code = ResourceExhausted", or writes are throttled

[Error] [director] [target-<target id>] [gcp-logs-high-volume] Sender worker 0 Finalize failed on flush for target "gcp-logs-high-volume": failed to send logs to Google Cloud Logging after 0 retries: saw 1 errors; last: rpc error: code = ResourceExhausted desc = ...

Cause: a Cloud Logging quota for the project is exhausted. The description typically names the quota metric that was exceeded, usually the write-request rate or the ingestion rate.

Fix: request an increase for the named quota in the console, on the quotas page of the Cloud Logging API for that project. You can also send fewer, larger requests by raising batch_size toward its ceiling of 10000. Values above the ceiling are clamped and logged as a warning:

batch_size 20000 exceeds the 10000-record ceiling and was clamped to it; lower the value in the config to silence this

No data is lost. The batch is retried until the service accepts it, so the queue grows for as long as the quota is exhausted.

"flush operation timed out", or network and certificate failures

[Error] [director] [target-<target id>] [gcp-logs] Sender worker 1 execute() failed for gcp-logs: target broken: failed to finalize target cache: flush operation timed out: context deadline exceeded

Cause: the flush did not finish within timeout seconds, 30 by default. Nearly every network problem reaches you in this form, because a failing connection is retried internally for far longer than timeout before any network error is reported. The underlying cause is typically a DNS failure, a firewall blocking logging.googleapis.com:443, or a TLS-inspecting proxy, which typically reports x509: certificate signed by unknown authority. Those descriptions are written to the Director process output, prefixed logging client:, rather than to the target log, so look there when the timeout alone does not tell you enough.

Fix:

• Allow outbound HTTPS on port 443 from the Director host to logging.googleapis.com and to oauth2.googleapis.com.
• If the host reaches the internet through a proxy, set HTTPS_PROXY in the environment of the Director service. This target has no proxy property of its own.
• For an intercepting proxy, install its certificate authority in the Director host's own trust store. There is no option on this target to trust a custom CA or to skip verification.
• With authentication.method: auto on a host that is not a Google Cloud instance, the metadata service is probed once per start. Set NO_GCE_CHECK=true in the Director environment to skip that probe.
• Raise timeout, or lower batch_size, when the link is simply slow rather than blocked.

No data is lost. The batch is retried until it succeeds.

warning

A flush that times out can still complete afterwards. When it does, the records it carried are delivered again by the retry, so duplicate entries in Cloud Logging are possible after a period of timeouts.

Other errors reported on flush

Error text	Cause and fix
rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: auth: cannot fetch token: 400	The service-account key was deleted, rotated, or disabled, or the Director host clock is skewed. Issue a new key and update credentials, and check time synchronization on the host
rpc error: code = InvalidArgument desc = ...	The entry was rejected as malformed. Typically log_name holds characters Cloud Logging does not accept, a label is outside the accepted limits, or a record carries no usable timestamp. Correct log_name and labels first. This one is retried like any other failure, so a single rejected batch is redelivered indefinitely and holds up everything queued behind it
item size exceeds bundle byte limit	One record is too large for a single write. Split or trim it earlier in the pipeline
panic during cloud logging flush: ...	Raise it with VirtualMetric support with the full log line
failed to close Google Cloud Logging client: ...	Reported while the target is shutting down or restarting, and it repeats the cause of the last failed flush. Fix that cause. There is nothing to do about the close itself

Configuration errors that stop the target from starting

The first three below are reported as ValidateConfig failed for target "gcp-logs": ... and the rest as Failed to reinitialize target "gcp-logs" (attempt N). Reason: .... Both repeat until you change the configuration. Nothing is sent while any of them lasts, and nothing is lost.

Reason text	Fix
project_id is required for google cloud logging target	Set project_id
log_name is required for google cloud logging target	Set log_name
credentials are required when authentication method is manual or secret	Set credentials, or use authentication.method: auto
invalid authentication method: Manual	authentication.method must be exactly auto, manual, or secret, in lower case. A capitalized or otherwise different value is not rejected when you save it, only here
batch_size must be greater than 0	Remove batch_size to use the default of 1000, or set a positive value
Google Cloud Logging logger not initialized	The target was used before it had finished starting. It clears on the next attempt

The target is healthy but nothing arrives, or the entries are not what you configured

Nothing fails in these cases, so there is no error to search for. Check them in order.

1. max_retries is 1 or more. After a flush that has already failed, a retry can report success without those records having been delivered. The target then counts the batch as sent and the payloads are acknowledged, so they are gone. The events-out counter rises and nothing appears in Logs Explorer. Keep max_retries at its default of 0, which reports the failure and keeps the data queued.

2. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No counter moves, nothing is written, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well:

   Log sending is disabled for this target (gcp-logs). Logs will be processed by the pipeline but will not be sent to the target.

3. The entries have the wrong severity. severity is matched exactly, in upper case. A value such as warning, Warn, or ERR is not recognized and no warning is logged. Those entries are written with DEFAULT severity, so a Logs Explorer filter on a severity level does not return them. Use one of the nine values listed under Severity Levels above.

4. The entries have no labels. labels must be a YAML mapping. A string or a list is accepted without a warning and produces entries with no labels at all, so a filter on a label returns nothing.

5. The batches are not the size you set. batch_size: 0, or a value that is not a number, is silently replaced with the default of 1000. A value above 10000 is clamped, with a warning in the log.