Google Cloud Pub/Sub

Google Cloud Message Queue

Synopsis

Creates a target that publishes messages to Google Cloud Pub/Sub topics with support for batch processing, message ordering, and service account authentication. Provides reliable message delivery to Google Cloud Pub/Sub for event-driven architectures and distributed systems.

Schema

- name: <string>
  description: <string>
  type: gcppubsub
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    topic_id: <string>
    allow_auto_topic_creation: <boolean>
    credentials_file: <string>
    credentials_json: <string>
    ordering_key: <string>
    max_messages: <numeric>
    max_bytes: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be gcppubsub
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
project_id	Y	-	Google Cloud project ID
topic_id	Y	-	Pub/Sub topic ID
allow_auto_topic_creation	N	false	Create the topic if it does not exist. Requires roles/pubsub.editor in place of roles/pubsub.publisher. See Topic Management below
credentials_file	N	-	Path to service account JSON key file (file must exist if specified)
credentials_json	N	-	Service account JSON key as inline string or environment variable expansion (must be valid JSON if specified)

note

If neither credentials_file nor credentials_json is provided, the target uses Application Default Credentials (ADC) — GOOGLE_APPLICATION_CREDENTIALS environment variable, gcloud CLI credentials, or GCE/GKE metadata service. The service account needs roles/pubsub.publisher on a pre-created topic. If allow_auto_topic_creation is enabled, use roles/pubsub.editor instead. See IAM Permissions below.

Message Configuration

Field	Required	Default	Description
ordering_key	N	-	Fixed ordering key applied to every message. Not a template, so it cannot vary per record
max_messages	N	1000	Maximum number of messages per batch
max_bytes	N	10485760	Maximum batch size in bytes (10 MB)
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Google Cloud Pub/Sub target publishes messages to Pub/Sub topics for asynchronous, scalable message delivery. It supports batch processing for optimal performance and message ordering for maintaining event sequence.

Authentication

The target supports three authentication modes, evaluated in this order:

1. credentials_file — Path to a service account JSON key file. The file is validated for existence during configuration.
2. credentials_json — Service account JSON key as an inline string (typically supplied via environment variable expansion). The JSON is validated during configuration.
3. Application Default Credentials (ADC) — Used automatically when neither of the above is provided. Resolves via the GOOGLE_APPLICATION_CREDENTIALS environment variable, gcloud CLI credentials, or GCE/GKE compute metadata service.

IAM Permissions

What the service account needs depends on whether the target may create its own topic:

Setup	Role	Minimum permissions
Pre-created topic (default)	roles/pubsub.publisher	pubsub.topics.publish
Pre-created topic, with early validation	+ roles/pubsub.viewer	+ pubsub.topics.get
Auto-creation enabled	roles/pubsub.editor	+ pubsub.topics.create

roles/pubsub.viewer is genuinely optional. It buys a clear error at startup when topic_id is wrong, instead of the first publish failing with NotFound.

note

The startup probe is advisory. The target reads the topic's metadata before publishing, but roles/pubsub.publisher grants pubsub.topics.publish and not pubsub.topics.get — so a correctly configured publisher-only credential cannot perform that read, and the probe comes back PermissionDenied. Rather than demand a broader grant than publishing actually needs, the target logs a warning and carries on. Only a definitive NotFound decides whether the topic is created or startup fails. If the topic turns out to be absent after an unverified start, the first publish fails with a NotFound naming it.

Topic Management

The target does not create its topic by default. At startup it reads the metadata of the topic named by topic_id, in the project named by project_id, and acts on the answer:

Topic state	allow_auto_topic_creation	Result
Exists	either	Publishes
Absent	true	Topic is created, then publishes
Absent	false or unset	Initialization fails, naming the topic and the key
Could not be read, PermissionDenied	either	Warning logged; startup continues and publishing is attempted
Could not be read, any other error	either	Initialization fails with failed to verify topic

In the web interface the control is Allow auto topic creation, in the Pub/Sub Configuration step beside Topic ID, described there as "Create the topic if it does not exist. Requires the roles/pubsub.editor IAM role."

Behavior change

Auto-creation used to be unconditional. A target that relied on it will now fail to start unless allow_auto_topic_creation is set to true, or the topic is created in advance.

Why the default is off. An auto-created topic has no subscription attached, and Pub/Sub discards every message published to a topic that has no subscription. A typo in topic_id therefore used to create a brand-new empty topic and blackhole everything this target sent, while the target went on reporting healthy. Failing at startup surfaces the typo instead. If you do enable auto-creation, attach a subscription yourself afterwards.

Message Ordering

When an ordering_key is set, it is applied unchanged to every message the target publishes. Pub/Sub delivers messages that share an ordering key in the order they were published, so a single key keeps the whole stream from this target in sequence.

The value is a fixed string, not a template. It cannot vary per record, so it cannot carry a device ID or any other field value. The subscription must also have message ordering enabled for the order to be honoured.

note

One ordering key for the whole target means every message shares one publish rate limit, and one failed publish pauses the rest until the error clears. Leave ordering_key unset unless you need ordering.

Message Attributes

The target automatically adds the following attributes to each message:

• device_id - Source device identifier
• device_type - Type of source device
• device_name - Name of source device

These attributes can be used for message filtering and routing in subscriptions.

Batch Processing

Messages are published in batches for optimal throughput. The target accumulates messages until the batch size limit is reached or during finalization. Google Cloud Pub/Sub supports up to 10 MB per batch.

At-Least-Once Delivery

Pub/Sub guarantees at-least-once delivery. Messages may be delivered more than once in case of network issues or subscriber failures. Design your message handlers to be idempotent.

Message Retention

Messages are retained for 7 days by default. You can configure retention periods from 10 minutes to 7 days in the topic settings.

Dead Letter Topics

Google Cloud Pub/Sub supports dead letter topics for messages that cannot be processed after a configured number of delivery attempts. Configure this in the subscription settings.

Examples

The following are commonly used configuration types.

Basic with File Credentials

Creating a basic Pub/Sub target with credentials file...	- name: basic_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "application-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 1000
Target publishes JSON messages to Pub/Sub topic...	{ "timestamp": "2024-01-15T10:30:00Z", "host": "server01", "message": "Application started successfully", "severity": "info" }

With JSON Credentials

Using credentials as JSON string (useful for secrets management)...

- name: json_creds_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "security-events" credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}" max_messages: 500

With Message Ordering

Using ordering key to maintain message sequence...

- name: ordered_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "ordered-events" credentials_file: "/path/to/service-account-key.json" ordering_key: "device-001" max_messages: 250

High-Throughput Configuration

Optimizing for high-volume message publishing...

- name: high_volume_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "high-volume-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 2000 max_bytes: 20971520

Multiple Topics

Publishing to different topics for different log types...

- name: application_logs_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "application-logs" credentials_file: "/path/to/service-account-key.json" - name: error_logs_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "error-logs" credentials_file: "/path/to/service-account-key.json"

Field Normalization

Using field normalization for standard format...

- name: normalized_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "normalized-logs" credentials_file: "/path/to/service-account-key.json" field_format: "ecs" max_messages: 1000

Pipeline Processing

Applying post-processing pipelines before publishing...

- name: pipeline_pubsub type: gcppubsub pipelines: - format_timestamp - add_metadata - validate_schema properties: project_id: "my-project-id" topic_id: "processed-events" credentials_file: "/path/to/service-account-key.json" max_messages: 750

Scheduled Batching

Configuration with scheduled batch delivery...

- name: scheduled_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "scheduled-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 1000 interval: "5m"

Debug Configuration

Configuration with debugging enabled...

- name: debug_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "test-logs" credentials_file: "/path/to/service-account-key.json" debug: status: true dont_send_logs: true

Production Configuration

Configuration for production with optimal settings...

- name: production_pubsub type: gcppubsub pipelines: - checkpoint properties: project_id: "production-project" topic_id: "production-logs" credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}" ordering_key: "production-cluster" max_messages: 1500 max_bytes: 15728640 field_format: "ecs"

Troubleshooting

This section covers the errors you are most likely to see with the gcppubsub target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

Errors that come back from Pub/Sub arrive inside the remote call framing rpc error: code = ... desc = .... Read the code first. The description after desc = is written by the service and its wording changes over time, so the entries below are grouped by code rather than by sentence.

Which permission is missing?

At startup the target reads the metadata of the topic named by topic_id, and creates it only when the topic is definitively absent and allow_auto_topic_creation is true. In the default configuration publishing is what your role is checked against; pubsub.topics.create is reached only with auto-creation enabled.

Grant roles/pubsub.publisher on the topic for the default setup, and add roles/pubsub.viewer if you want a clear startup error on a mistyped topic_id rather than a NotFound on the first publish. If the target may create the topic, use roles/pubsub.editor instead — it covers all three permissions on its own. roles/pubsub.admin also works but grants far more than the target uses.

Error text	Missing role or permission	Granted on
failed to publish message: rpc error: code = PermissionDenied	pubsub.topics.publish, included in roles/pubsub.publisher and in roles/pubsub.editor	The topic named by topic_id, or the project
failed to create topic: rpc error: code = PermissionDenied	pubsub.topics.create, included in roles/pubsub.editor. Only reachable with allow_auto_topic_creation: true	The project named by project_id
failed to create topic: rpc error: code = PermissionDenied whose description typically mentions the API not being used or being disabled	No role. The Pub/Sub API is not enabled	The project named by project_id
failed to create pub/sub client, or any code = Unauthenticated	No role. The credential itself was refused before permissions were reached	See the credential entries below

note

A startup warning that the topic could not be verified is expected with a publisher-only credential, and is not a failure:

Cannot verify topic "<topic id>": the credential lacks pubsub.topics.get. Proceeding without verification.

roles/pubsub.publisher does not grant pubsub.topics.get, so the probe cannot read topic metadata. The target proceeds; if the topic really is absent, the first publish fails with a NotFound naming it. Add roles/pubsub.viewer if you would rather have that caught at startup.

"failed to create topic" with code = PermissionDenied

[Error] [director] [target-<target id>] [basic_pubsub] Failed to reinitialize target "basic_pubsub" (attempt 4). Reason: failed to create topic: rpc error: code = PermissionDenied desc = ...

Cause: the service account authenticated, but it may not create a topic in the project. This is reachable only with allow_auto_topic_creation: true — with the default the target never calls create. A description that typically mentions the API not being used in the project, or being disabled, points at the Pub/Sub API rather than at the role.

Fix: either grant roles/pubsub.editor on the project in project_id to the service account in your key file, from IAM > Grant access in the Google Cloud console, or pre-create the topic and leave allow_auto_topic_creation off, which needs only roles/pubsub.publisher. Then confirm the Pub/Sub API is enabled on that project. Role changes need a little time to take effect, and Director keeps retrying, so no restart is needed.

Nothing is sent while this lasts. Incoming data stays queued and is delivered once the target starts.

"failed to create pub/sub client"

[Error] [director] [target-<target id>] [basic_pubsub] Failed to reinitialize target "basic_pubsub" (attempt 3). Reason: failed to create pub/sub client: pubsub(publisher): credentials: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

Cause: the credential could not be resolved or read, so no call was ever made. The text after pubsub(publisher): names the exact problem.

Text after pubsub(publisher):	What it means	Fix
credentials: could not find default credentials. ...	Neither credentials_file nor credentials_json is set, so Application Default Credentials were used, and this host has none	Set credentials_file, or export GOOGLE_APPLICATION_CREDENTIALS for the Director service, or run Director on a Google Cloud instance with an attached service account
open /path/to/service-account-key.json: permission denied, or Access is denied. on Windows	The key file is there, but the user the Director service runs as may not read it	Give that user read access to the file
credentials: unsupported unidentified file type	The value is valid JSON but is not a Google credential. An OAuth client secret file, or any other JSON document, looks like this	Download a service account key and use that file or its contents
failed to parse private key. Tried PKCS8, PKCS1, and EC formats. Errors: ...	The private key inside the JSON is damaged. Line breaks lost or doubled on the way through a secret store, an environment variable, or a YAML block do this	Use credentials_file with the key file exactly as it was downloaded

Fix: correct the credential as shown above. Nothing is sent while this lasts, and it is retried until you fix the cause. A repaired key file is picked up on the next retry, so Director does not have to be restarted.

A code = Unauthenticated error later on means the credential was accepted locally but refused by Google. The usual causes are a key that was deleted or disabled, a service account that no longer exists, a key belonging to a different project, and a host clock that has drifted far enough for the signed token to be rejected. Issue a new key for a live service account, and check time synchronisation on the Director host.

"credentials_json is not valid JSON" or "credentials_file does not exist"

[Error] [director] [target-<target id>] [json_creds_pubsub] ValidateConfig failed for target "json_creds_pubsub": credentials_json is not valid JSON: invalid character '$' looking for beginning of value

Cause: invalid character '$' means the value still holds the literal ${GCP_SERVICE_ACCOUNT_JSON} text. The variable was never expanded, usually because it is set in the shell you tested from rather than in the environment of the Director service. Other JSON errors mean the key was damaged in transit, for example by shell quoting, or that a file path was pasted where the key contents belong. credentials_file does not exist names a path that is not present on this Director host. A relative path is resolved against the service working directory, so give an absolute one.

Fix: prefer credentials_file with an absolute path to the key file on the host. If you keep credentials_json, set the variable in the service environment and restart the service so it is picked up. Only JSON syntax is checked here, so a document that parses but is not a service account key passes this check and fails at startup instead, with credentials: unsupported unidentified file type.

The configuration is re-checked about every 30 seconds and nothing is sent until it is valid, but no data is lost while you correct it.

warning

If credentials_json resolves to an empty string, you get no error at all. The target treats the field as unset and falls back to Application Default Credentials. On a Google Cloud instance that means it publishes as the instance's own service account, which is a different identity with different permissions than the one you configured.

"topic ... does not exist" and auto-creation is off

[Error] [director] [target-<target id>] [basic_pubsub] Failed to reinitialize target "basic_pubsub" (attempt 1). Reason: topic "<topic id>" does not exist in project "<project id>" and allow_auto_topic_creation is disabled

Cause: the startup probe came back with a definitive NotFound for topic_id, and allow_auto_topic_creation is false or unset, which is the default. Earlier versions created the topic at this point; the target now refuses to, so that a mistyped topic_id surfaces here instead of producing a new empty topic that silently discards everything published to it.

Fix: check topic_id against the topic in the console first — a typo is the likelier cause than a genuinely missing topic. Then either create the topic and attach a subscription to it, or set allow_auto_topic_creation: true and grant roles/pubsub.editor. If you enable auto-creation, attach a subscription afterwards yourself: an auto-created topic has none, and Pub/Sub discards messages published to a topic with no subscription.

Nothing is sent while this lasts. Incoming data stays queued and is delivered once the target starts.

"failed to create topic" with code = NotFound or code = InvalidArgument

[Error] [director] [target-<target id>] [basic_pubsub] Failed to reinitialize target "basic_pubsub" (attempt 1). Reason: failed to create topic: rpc error: code = NotFound desc = ...

Cause: NotFound means the project in project_id does not exist, or this service account cannot see it. InvalidArgument means the name itself could not be used, and the description typically quotes back the resource name it was given, which is the quickest way to see what was built from your two fields.

Fix: set project_id to the project ID, which is the lowercase identifier shown in the console, not the project number and not the display name. Set topic_id to the bare topic ID, for example my-topic. It must be 3 to 255 characters, start with a letter, contain only letters, digits and the characters -, _, ., ~, + and %, and must not start with goog. A full projects/my-project/topics/my-topic path is not accepted, because Director builds that path itself from project_id and topic_id, and a slash in topic_id therefore produces an invalid name.

Both fail the same way on every retry until the configuration changes. Nothing is sent in the meantime, and nothing is lost.

"code = Unavailable", or a startup error that appears about a minute late

[Error] [director] [target-<target id>] [basic_pubsub] Failed to reinitialize target "basic_pubsub" (attempt 12). Reason: failed to create topic: rpc error: code = Unavailable desc = ...

Cause: Director could not reach pubsub.googleapis.com on port 443. This one is retried inside the client for up to a minute before it is logged, so the first line appears about 60 seconds after the target starts, and roughly once a minute after that. The description typically names the step that failed: no such host for a DNS problem, i/o timeout or connection refused for a blocked port, and certificate signed by unknown authority when a TLS-inspecting proxy presents a certificate the host does not trust.

Fix:

• Allow outbound TCP 443 from the Director host to pubsub.googleapis.com.
• Add oauth2.googleapis.com when the credential is a gcloud user credential picked up through Application Default Credentials. A service account key file does not need it.
• Set HTTPS_PROXY and NO_PROXY in the environment of the Director service, not only in your own shell.
• For a TLS-inspecting proxy, install its CA certificate in the Director host's trust store. This target has no setting for a custom CA and none for skipping verification.

Nothing is sent while this lasts, and it is retried until the host can reach the service.

note

A PUBSUB_EMULATOR_HOST value left over in the service environment redirects this target away from Google Cloud entirely. It usually surfaces as the same Unavailable error, against an address where nothing is listening.

"failed to publish message" with code = DeadlineExceeded or code = ResourceExhausted

[Error] [director] [target-<target id>] [production_pubsub] Sender worker 1 execute() failed for <payload>: target broken: failed to finalize target cache: failed to publish message: rpc error: code = DeadlineExceeded desc = ...

Cause: the publish call was retried inside the client for up to a minute and then gave up, so this line describes something that started about a minute earlier. ResourceExhausted typically carries a quota description. DeadlineExceeded, whose description is typically context deadline exceeded, is what throttling usually looks like by the time it reaches the log, because the retries consume the deadline before the last one is reported. An ordering_key makes this more likely: every message the target publishes carries the same key, so the whole target shares one ordering key and the publish rate limit that comes with it.

Fix: remove ordering_key unless you need ordering. Request more publisher quota for the project. Set interval or cron so bursts are spread out, as described under Scheduling and Pool Behavior.

Nothing is lost. The batch is redelivered until it is accepted. Messages that did reach the topic before the failure are published again on the retry, so subscribers see duplicates.

"paused due to previous error"

[Error] [director] [target-<target id>] [ordered_pubsub] Sender worker 2 execute() failed for <payload>: target broken: failed to finalize target cache: failed to publish message: pubsub: Publishing for ordering key, device-001, paused due to previous error. ...

Cause: this is collateral, not a new fault. With ordering_key set, one failed publish pauses that key, and every message queued behind it fails this way, in all workers at once, because they share a single publisher and a single key. A burst of these lines always follows some other publish error.

Fix: fix the error in the line just before the burst. Director resumes the key by itself once the underlying cause clears, so there is nothing to reset by hand.

Nothing is lost. The affected batches are redelivered in full, which produces duplicates for the messages that had already landed.

"item size exceeds bundle byte limit"

[Error] [director] [target-<target id>] [high_volume_pubsub] Sender worker 3 execute() failed for <payload>: target broken: failed to finalize target cache: failed to publish message: item size exceeds bundle byte limit

Cause: one record is too large for a single publish request. The ceiling is 10,000,000 bytes for the encoded message, counting the attributes and the ordering key as well as the data, minus the length of the topic path. The default max_bytes of 10485760 is higher than that ceiling, so a record of roughly 9.9 MB or more passes Director's own size check and is then refused when it is published. The max_bytes: 20971520 in the High-Throughput example above widens that gap further.

Fix: keep records well under 9.9 MB, and split or trim oversized ones in the pipeline before they reach the target. Setting max_bytes to 9900000 or lower moves the refusal back to Director, where the batch boundary handles it.

warning

The record is never dropped. It is retried every few seconds for as long as it stays in the queue, and it holds up everything queued behind it, so treat it as urgent. Records batched ahead of it were delivered normally.

"batch limit exceeded" repeating in the logs

[Error] [director] [target-<target id>] [basic_pubsub] Sender worker 2 execute() failed for <payload>: batch limit exceeded

Cause: a single record is larger than max_bytes on its own. Reaching max_messages or max_bytes normally just closes the current batch and is never logged. It only becomes an error when the record does not fit even into an empty batch.

Fix: raise max_bytes, staying under the 10,000,000 byte publish ceiling described above, or shrink the record in the pipeline. The data impact is the same as the previous entry: the record is retried rather than dropped, and it blocks the payloads behind it.

Configuration errors that stop the target from starting

These are reported before any connection is attempted. The configuration is re-checked about every 30 seconds until it is valid, and nothing is sent in the meantime.

Reason text	Fix
project_id is required for gcp pub/sub target	Set project_id to the project ID
topic_id is required for gcp pub/sub target	Set topic_id to the bare topic ID
max_messages must be greater than 0 for gcp pub/sub target	Remove the field to use the default of 1000, or set a positive number
max_bytes must be greater than 0 for gcp pub/sub target	Remove the field to use the default, or set a positive number

The target is healthy but nothing arrives in the topic

Nothing fails here, so there is no error to search for. Check the following in order.

1. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is published. Nothing is counted as dropped, and the target reports healthy, so the delivery counters make it look as though the data went out. The only trace is one line at startup, and only when debug.status is enabled as well. Remove both flags when you have finished testing. The Debug Configuration example above turns this on.

   Log sending is disabled for this target (basic_pubsub). Logs will be processed by the pipeline but will not be sent to the target.

2. The messages went to another identity's project. If credentials_json resolved to an empty string, the target fell back to Application Default Credentials and published as whatever identity the host carries. Confirm which service account is in use, and which project owns the topic that is actually receiving the data.

3. A post-processor pipeline is failing. Anything listed under pipelines runs before delivery. A record the pipeline rejects the same way every time is given up on after four attempts and counted as dropped, not delivered. Check the target's dropped counters, then fix the pipeline.

4. The topic has no subscription. Pub/Sub discards every message published to a topic that has no subscription attached, and the publish still succeeds, so the target reports healthy and its delivery counters rise. This is the usual outcome of a topic created by allow_auto_topic_creation, which attaches none. Confirm the topic has at least one subscription, and that it is the topic you meant — a mistyped topic_id with auto-creation enabled produces exactly this.

Subscribers see the same message more than once

Cause: Pub/Sub delivers at least once by design, and Director adds one more source of duplicates. When a batch fails part of the way through, the whole batch is redelivered, so every message that had already been accepted is published a second time. The target's delivered count grows for both copies, so it can read higher than the number of distinct records you sent.

Fix: there is nothing to change in the target. Make subscribers idempotent, for example by deduplicating on a field your pipeline sets. Expect a run of duplicates after any of the publish errors above, and the most after a throttling burst with ordering_key set.