IBM Cloud Logs

IBM Cloud Target

Synopsis

The IBM Cloud Logs target forwards telemetry events to IBM Cloud Logs using the Logs API singles endpoint. Events are batched and sent with configurable application context, subsystem categorization, and severity levels.

Schema

- name: <string>
  description: <string>
  type: ibmcloudlogs
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    instance_id: <string>
    region: <string>
    authentication_method: <string>
    iam_token: <string>
    iam_token_secret: <string>
    application_name: <string>
    subsystem_name: <string>
    computer_name: <string>
    default_severity: <integer>
    use_timestamp: <boolean>
    use_hires_timestamp: <boolean>
    batch_size: <integer>
    timeout: <integer>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be ibmcloudlogs
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

IBM Cloud Logs Connection

Field	Required	Default	Description
instance_id	Y	-	IBM Cloud Logs instance ID
region	Y	-	IBM Cloud region. See Valid Regions below
authentication_method	N	token	Authentication method: token, secret
iam_token	Y*	-	IBM Cloud IAM Bearer token
iam_token_secret	Y*	-	Environment variable name containing IAM token

* = Conditionally required. iam_token when authentication_method: token; iam_token_secret when authentication_method: secret.

Log Configuration

Field	Required	Default	Description
application_name	Y	-	Application name for log categorization
subsystem_name	Y	-	Subsystem name for log categorization
computer_name	N	-	Computer/host name for log source identification
default_severity	N	1	Default severity level (1-6). 1 = Debug. See Severity Levels below
use_timestamp	N	false	Use event timestamp instead of current time
use_hires_timestamp	N	false	Use high-resolution (nanosecond) timestamp

Batch Configuration

Field	Required	Default	Description
batch_size	N	1000	Maximum events per batch
timeout	N	30	Request timeout in seconds

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Valid Regions

IBM Cloud Logs is available in the following regions:

Region Code	Region Name
us-south	US South (Dallas)
us-east	US East (Washington DC)
eu-gb	United Kingdom (London)
eu-de	Germany (Frankfurt)
eu-es	Spain (Madrid)
jp-tok	Japan (Tokyo)
jp-osa	Japan (Osaka)
au-syd	Australia (Sydney)
ca-tor	Canada (Toronto)
br-sao	Brazil (São Paulo)

Severity Levels

IBM Cloud Logs uses numeric severity levels:

Level	Name	Description
1	Debug	Debug or trace information
2	Verbose	Verbose informational messages
3	Info	Informational messages
4	Warn	Warning events
5	Error	Error events
6	Critical	Critical events requiring immediate action

Severity Handling:

• Events with severity field use that value if valid (1-6)
• Events without severity field use default_severity
• Invalid severity values default to default_severity

Authentication Methods

Token (Default):

• Use iam_token field with Bearer token directly in configuration
• Token automatically prefixed with "Bearer " if not already present
• Simpler for development and testing

Secret:

• Use iam_token_secret field with environment variable name
• More secure for production deployments
• Environment variable must be set before starting DataStream

IAM Token Requirements

IBM Cloud Logs requires a valid IAM Bearer token for authentication. Generate tokens using the IBM Cloud CLI or API. Director adds the Bearer prefix only when your value does not already start with it, then sends the token as configured. It never renews or exchanges the token, so supply a fresh one before the current one expires.

IAM Permissions

The IAM identity (service ID or user) used to generate the Bearer token requires the following IBM Cloud IAM role:

IBM IAM Role	Service	IAM Action	Purpose
Sender	IBM Cloud Logs	logs.data.send	Ingest logs via the /logs/v1/singles REST endpoint

The Sender role (underlying IAM action: logs.data.send) is the minimum required role. No read, management, or administrative roles are needed since the target only performs log ingestion (write-only).

note

IBM Cloud Logs accepts up to 2 MB per request, which is approximately 3,000 medium-sized log entries.

Endpoint Construction

Automatic Endpoint Building:

• Endpoint format: https://{instance_id}.ingress.{region}.logs.cloud.ibm.com/logs/v1/singles
• Example: https://abc123.ingress.us-south.logs.cloud.ibm.com/logs/v1/singles
• Instance ID and region are validated during configuration

Event Structure

JSON Parsing:

• Events with valid JSON message are parsed and sent as structured data
• Non-JSON messages are sent as {"text": "message"} objects
• Supports nested JSON structures and complex data types

Application Context:

• applicationName: Required field for log categorization
• subsystemName: Required field for subsystem identification
• computerName: Optional field for source host identification

Timestamp Handling:

• Default: Current timestamp when event is sent
• use_timestamp: true: Use event's original timestamp
• use_hires_timestamp: true: Include nanosecond precision

Performance Considerations

Batch Processing:

• Events are buffered until batch_size is reached
• Flush occurs on batch limit or during finalization
• Larger batches reduce API calls but increase latency
• Maximum recommended batch size: 1000 events

Connection Pooling:

• HTTP client maintains connection pool
• Maximum 100 idle connections total
• Maximum 10 idle connections per host
• 90-second idle connection timeout

Retry Logic:

• Failed sends are retried based on sender configuration
• HTTP errors include response body for troubleshooting
• Check IBM Cloud Logs service status for API issues

Batch Size Limits

IBM Cloud Logs API has limits on batch size and request payload. Configure batch_size appropriately for your event size to avoid API rejections.

Error Handling

Authentication Failures:

• HTTP 401: Invalid or expired IAM token
• Supply a fresh token. In secret mode, update the environment variable and restart the service
• HTTP 403: The identity behind the token lacks the Sender role on the instance
• See Troubleshooting below

API Errors:

• HTTP 400: Malformed request or invalid event structure
• HTTP 500: IBM Cloud Logs service error
• Error responses include detailed message for troubleshooting

Validation Errors:

• Invalid region codes are rejected during configuration validation
• Invalid severity levels default to default_severity
• Missing required fields (instance_id, application_name, subsystem_name) prevent target initialization

Examples

Basic Configuration

Sending logs to IBM Cloud Logs using token authentication...

targets: - name: ibm-logs type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: datastream subsystem_name: telemetry

With Secret Authentication

Using environment variable for secure IAM token storage...

targets: - name: ibm-logs-secure type: ibmcloudlogs properties: instance_id: xyz789abc123 region: eu-gb authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: security subsystem_name: audit computer_name: production-server

With Custom Severity

Setting default severity to Warning for important events...

targets: - name: ibm-logs-warnings type: ibmcloudlogs properties: instance_id: abc123def456 region: us-east iam_token: "${IBM_IAM_TOKEN}" application_name: monitoring subsystem_name: alerts default_severity: 4 use_timestamp: true

High-Volume Configuration

Optimizing for high-volume log ingestion with larger batches...

targets: - name: ibm-logs-high-volume type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: streaming subsystem_name: events batch_size: 1000 timeout: 30 use_timestamp: true use_hires_timestamp: true

Multi-Region Configuration

Sending logs to different IBM Cloud regions for geographic distribution...

targets: - name: ibm-logs-us type: ibmcloudlogs properties: instance_id: us123abc456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: us-region - name: ibm-logs-eu type: ibmcloudlogs properties: instance_id: eu456def789 region: eu-de iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: eu-region

With Normalization

Applying ECS normalization before sending to IBM Cloud Logs...

targets: - name: ibm-logs-normalized type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: security subsystem_name: normalized field_format: ECS default_severity: 3

Production Configuration

Production-ready configuration with secret authentication, batch optimization, and high-resolution timestamps...

targets: - name: ibm-logs-production type: ibmcloudlogs properties: instance_id: prod123abc456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: production-datastream subsystem_name: telemetry-processing computer_name: datastream-director-01 default_severity: 3 use_timestamp: true use_hires_timestamp: true batch_size: 1000 timeout: 30 field_format: ASIM

Troubleshooting

This section covers the errors you are most likely to see with the ibmcloudlogs target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

Two facts shape most of the entries below. The target makes no network call while it starts, so a wrong token, region or instance ID stays invisible until the first batch is flushed, and the connection status looks healthy until then. And only the payload rejections described below ever discard data. Every other failure is retried until you fix it.

What the service needs

Prerequisite	Where you set it	Error when it is missing or wrong
An IAM Bearer token that is still valid at the moment of the request	iam_token, or the environment variable named by iam_token_secret	ibm cloud logs api returned status 401
The Sender role, IAM action logs.data.send, on the Cloud Logs instance, held by the identity that minted the token	IBM Cloud IAM	ibm cloud logs api returned status 403
The code of the region your instance actually runs in	region	invalid region: ... at configuration time, or a name resolution failure at the first flush

"ibm cloud logs api returned status 401"

[Error] [director] [target-<target id>] [ibm-logs] Sender worker 3 execute() failed for ibm-logs: ibm cloud logs api returned status 401: <response body>

The text after the status code is the response the service returned. It typically carries a short JSON message naming the reason.

Cause: the service refused the token. Director puts the value you configure into the Authorization header exactly as you wrote it, on every request, and never renews or exchanges it. IAM access tokens are short-lived, so a deployment that is delivering happily today fails with this status from the moment the configured token passes its expiry, and keeps failing until you supply a new one. Treat token rotation as scheduled work rather than something the target handles for you.

Three other mistakes produce the same status:

• The token was pasted with a leading or trailing space or a line break.
• The token already starts with a lower-case bearer . The Bearer prefix is added for you only when the value does not already begin with it in exactly that spelling, so bearer ... is sent as Bearer bearer ....
• The credential reference never resolved. See the next entry.

Fix: generate a fresh token and give it to the target. In token mode, update iam_token and save. The failure already has the target reinitializing, so the new value is picked up on the next attempt and no restart is needed. In secret mode, set the environment variable in the service environment and restart Director, because a running process does not see a variable exported after it started.

No data is lost. The batch is redelivered about every 5 seconds until a request succeeds, so the queue grows for as long as the token is stale.

note

Rotate the token before it expires rather than after. Nothing is logged while a token is still valid, so the first sign of expiry is this error on live traffic.

"ibm cloud logs api returned status 403"

[Error] [director] [target-<target id>] [ibm-logs-production] Sender worker 2 Finalize failed on flush for target "ibm-logs-production": ibm cloud logs api returned status 403: <response body>

Cause: the token itself is accepted, but the identity behind it is not allowed to send to this instance. Either the Sender role was never granted, or the token was minted in a different IBM Cloud account from the one that owns the instance.

Fix: in the IBM Cloud console under Access (IAM), grant Sender on the Cloud Logs instance to the service ID or user that mints the token, then generate the token again so it carries the new authorization. Confirm you authenticated against the account that owns the instance.

No data is lost. Delivery is retried until the role is in place, so expect the queue to grow meanwhile.

"environment variable ... is not set or empty", or a token that never resolved

[Error] [director] [target-<target id>] [ibm-logs-secure] Failed to reinitialize target "ibm-logs-secure" (attempt 12). Reason: environment variable IBM_CLOUD_LOGS_TOKEN is not set or empty

Cause: with authentication_method: secret, iam_token_secret names an environment variable that Director reads from its own process environment. This message means the variable is absent there, or set to an empty value. Exporting it in an interactive shell is not enough. It has to be in the service unit, the container definition, or wherever the Director service takes its environment from.

The token method fails less visibly. A value such as iam_token: "${IBM_IAM_TOKEN}" is accepted even when IBM_IAM_TOKEN is not defined for the service. The unresolved text is then used as the credential, the target starts cleanly, and every request comes back as the 401 above.

Fix: for secret mode, define the variable in the service environment and restart Director. For token mode, confirm the variable exists for the service, or put the token into the field directly. A 401 that appears on the very first flush of a new target, rather than after a spell of working delivery, points at an unresolved reference rather than an expired token.

No data is lost either way. In secret mode the target never initializes and events wait in the queue. In token mode the requests fail and are retried.

"invalid region: ..."

[Error] [director] [target-<target id>] [ibm-logs] ValidateConfig failed for target "ibm-logs": invalid region: US-South

Cause: region is not one of the codes the target accepts. The comparison is exact and case-sensitive, so US-South, us_south and a value with a stray space are all refused. A region IBM Cloud Logs has added recently can be refused for the same reason.

Fix: copy the code from Valid Regions above, in lower case.

Nothing is sent while the configuration is invalid, and nothing is lost. The configuration is re-checked about every 30 seconds, so correcting the value is enough.

A region code that is valid but is not the region of your instance is a different problem. It passes validation, and the endpoint the target builds then points at a host that does not exist. You typically see a name resolution failure at the first flush:

[Error] [director] [target-<target id>] [ibm-logs] Sender worker 1 execute() failed for ibm-logs: failed to send request: Post "https://abc123def456.ingress.us-south.logs.cloud.ibm.com/logs/v1/singles": dial tcp: lookup abc123def456.ingress.us-south.logs.cloud.ibm.com: no such host

The same line appears when instance_id is the instance name instead of its GUID. Check both values against the instance details in the IBM Cloud console. Depending on what the wrong host resolves to, the service may instead answer with a not-found or an unauthorized status, so read those the same way. If instance_id holds a CRN, or anything else that cannot appear inside a host name, the URL cannot be built at all and the error reads failed to create request instead.

"record rejected by target ... status 413"

[Error] [director] [target-<target id>] [ibm-logs-high-volume] Sender worker 1 execute() failed for ibm-logs-high-volume: record rejected by target: ibm cloud logs api returned status 413: <response body>
[Error] [director] [target-<target id>] [ibm-logs-high-volume] Sender worker 1 deterministic failure for ibm-logs-high-volume after 4 attempts — dropping (giving up): record rejected by target: ibm cloud logs api returned status 413: <response body>

Cause: the request was larger than the service accepts. batch_size counts events, not bytes, so a batch of wide events can exceed the limit at any count. The service typically accepts 2 MB per request, roughly 3,000 medium-sized log records, and a full batch of wide events passes that easily.

Fix: lower batch_size until the rejection stops, halving it as a first step. Size the value against your largest events, not your typical ones, because one oversized request is enough to trigger this.

warning

This error loses data. After four delivery attempts the whole batch is dropped, up to batch_size events, not only the record that pushed the request over the limit. There is no split and retry.

"record rejected by target ... status 400" or "status 422"

[Error] [director] [target-<target id>] [ibm-logs] Sender worker 1 execute() failed for ibm-logs: record rejected by target: ibm cloud logs api returned status 400: <response body>

Cause: the service could not accept the records in the batch. The response body in the log line names the reason. A default_severity outside 1 to 6 is one trigger. Values above 6 are refused at configuration time, but a negative value passes validation and is then attached to every record. Field types the service does not expect are another trigger, usually from a pipeline that emits a number or an object where a string is expected.

Fix: read the response body in the log line first. Check default_severity, then application_name and subsystem_name, then the pipeline that produces the events.

This error loses data in the same way as 413. The whole batch is dropped after four delivery attempts, so a single malformed record costs its neighbors as well. These three statuses, 400, 413 and 422, are the only ones this target treats as permanent.

"ibm cloud logs api returned status 429"

[Error] [director] [target-<target id>] [ibm-logs] Sender worker 4 execute() failed for ibm-logs: ibm cloud logs api returned status 429: <response body>

Cause: the service is throttling your instance.

Fix: reduce how much you send per unit of time, spread the load across instances, or raise the quota on the plan. The target does not read a retry hint from the response, so it redelivers on the sender's own pacing of about 5 seconds.

No data is lost. Delivery latency grows while the throttling lasts.

"no such host", "connection refused", "i/o timeout", or a certificate error

[Error] [director] [target-<target id>] [ibm-logs] Sender worker 1 execute() failed for ibm-logs: failed to send request: Post "https://abc123def456.ingress.eu-de.logs.cloud.ibm.com/logs/v1/singles": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause and fix: the target reaches the ingress host over HTTPS on TCP 443, using the trust store of the Director host. Match the tail of the line against this table.

What the cause reads	Cause	Fix
connect: connection refused or i/o timeout	Outbound HTTPS to the ingress host is blocked	Allow TCP 443 from the Director host to *.logs.cloud.ibm.com
proxyconnect tcp: ...	HTTPS_PROXY is set for the Director service but the proxy cannot be reached	Correct the variable, or unset it if no proxy is needed
x509: certificate signed by unknown authority	A TLS-intercepting proxy presents a certificate the host does not trust. This target has no custom CA setting and no way to skip verification	Install the intercepting CA into the host trust store, or exclude *.logs.cloud.ibm.com from interception through NO_PROXY
context deadline exceeded	The exchange took longer than timeout seconds, 30 by default	Raise timeout, lower batch_size, or look at the network path
lookup ...: no such host	The host in the URL does not exist	Check region and instance_id, as described under the region entry above

No data is lost for any of these. Delivery is retried until the path works.

The target looks healthy but nothing arrives

Check these in order.

1. debug.dont_send_logs is enabled. Events are processed by the pipeline and then discarded before any request is made. Nothing is sent, no error is logged, and the delivered counter does not move. The line Log sending is disabled for this target (ibm-logs). Logs will be processed by the pipeline but will not be sent to the target. is written at startup only when debug.status is true as well, so with debug logging off the mode is completely silent. Set debug.dont_send_logs: false.

2. Your events carry a text or message key. When the top level of an event has either key, only that value is forwarded as the log text. Everything else is dropped, apart from the fields the target recognizes: applicationName, subsystemName, computerName, severity, category, className, methodName, threadId and the timestamps. Normalized output usually has a message field, so ECS and ASIM records arrive looking truncated. Rename or remove that key in a pipeline, or nest the payload so the top level has neither key. The whole object is then sent as the log text.

3. Timestamps are not the ones you expect. use_timestamp: true takes effect only when the event's timestamp is a number holding epoch milliseconds. An ISO 8601 string is ignored without any message, and the service typically stamps the arrival time instead. use_hires_timestamp: true needs hiResTimestamp present as a string, otherwise the field is left out. Convert both in the pipeline.

4. Every record has the same severity. A per-event severity is used only when it is a number from 1 to 6. A string such as "3", or a number outside the range, is ignored and default_severity is used instead.

5. batch_size or timeout is not the value you set. Both fall back to their defaults, 1000 events and 30 seconds, when the configured value is zero or is not a number, and nothing is logged about it. A negative timeout removes the request time limit altogether, so a stalled request can hold a worker indefinitely. Use positive integers.

6. The records are still in flight. Without an interval, the target sends each payload as it arrives, but the service adds its own indexing latency before records become searchable.