IBM Cloud Logs

IBM Cloud Target

Send processed telemetry data to IBM Cloud Logs service.

Synopsis

The IBM Cloud Logs target forwards telemetry events to IBM Cloud Logs using the Logs API singles endpoint. Events are batched and sent with configurable application context, subsystem categorization, and severity levels.

Schema

targets:
  - name: <string>
    type: ibmcloudlogs
    properties:
      instance_id: <string>
      region: <string>
      authentication_method: <string>
      iam_token: <string>
      iam_token_secret: <string>
      application_name: <string>
      subsystem_name: <string>
      computer_name: <string>
      default_severity: <integer>
      use_timestamp: <boolean>
      use_hires_timestamp: <boolean>
      batch_size: <integer>
      timeout: <integer>
      field_format: <string>
      debug:
        status: <boolean>
        dont_send_logs: <boolean>

Configuration

Base Target Fields

Field	Type	Required	Description
name	string	Y	Unique identifier for this target
description	string	N	Human-readable description
type	string	Y	Must be ibmcloudlogs
pipelines	array	N	Pipeline names to apply before sending
status	boolean	N	Enable (true) or disable (false) this target

IBM Cloud Logs Connection

Field	Type	Required	Description
instance_id	string	Y	IBM Cloud Logs instance ID
region	string	Y	IBM Cloud region (see Valid Regions)
authentication_method	string	N	Authentication method (token, secret). Default: token
iam_token	string	Y*	IBM Cloud IAM token (Bearer token)
iam_token_secret	string	Y*	Environment variable name containing IAM token

* iam_token required when authentication_method is token, iam_token_secret required when secret

Log Configuration

Field	Type	Required	Description
application_name	string	Y	Application name for log categorization
subsystem_name	string	Y	Subsystem name for log categorization
computer_name	string	N	Computer/host name for log source identification
default_severity	integer	N	Default severity level (1-6). Default: 1 (Debug)
use_timestamp	boolean	N	Use event timestamp instead of current time
use_hires_timestamp	boolean	N	Use high-resolution timestamp (nanoseconds)

Batch Configuration

Field	Type	Required	Description
batch_size	integer	N	Maximum events per batch. Default: 1000
timeout	integer	N	Request timeout in seconds. Default: 30

Normalization

Field	Type	Required	Description
field_format	string	N	Apply format normalization (ECS, ASIM, UDM)

Debug Options

Field	Type	Required	Description
debug.status	boolean	N	Enable debug logging for this target
debug.dont_send_logs	boolean	N	Log events without sending to IBM Cloud Logs

Details

Valid Regions

IBM Cloud Logs is available in the following regions:

Region Code	Region Name
us-south	US South (Dallas)
us-east	US East (Washington DC)
eu-gb	United Kingdom (London)
eu-de	Germany (Frankfurt)
eu-es	Spain (Madrid)
jp-tok	Japan (Tokyo)
jp-osa	Japan (Osaka)
au-syd	Australia (Sydney)
ca-tor	Canada (Toronto)
br-sao	Brazil (São Paulo)

Severity Levels

IBM Cloud Logs uses numeric severity levels:

Level	Name	Description
1	Debug	Debug or trace information
2	Verbose	Verbose informational messages
3	Info	Informational messages
4	Warn	Warning events
5	Error	Error events
6	Critical	Critical events requiring immediate action

Severity Handling:

• Events with severity field use that value if valid (1-6)
• Events without severity field use default_severity
• Invalid severity values default to default_severity

Authentication Methods

Token (Default):

• Use iam_token field with Bearer token directly in configuration
• Token automatically prefixed with "Bearer " if not already present
• Simpler for development and testing

Secret:

• Use iam_token_secret field with environment variable name
• More secure for production deployments
• Environment variable must be set before starting DataStream

IAM Token Requirements

IBM Cloud Logs requires a valid IAM Bearer token for authentication. Generate tokens using IBM Cloud CLI or API. Tokens expire and must be refreshed periodically.

Endpoint Construction

Automatic Endpoint Building:

• Endpoint format: https://{instance_id}.ingress.{region}.logs.cloud.ibm.com/logs/v1/singles
• Example: https://abc123.ingress.us-south.logs.cloud.ibm.com/logs/v1/singles
• Instance ID and region are validated during configuration

Event Structure

JSON Parsing:

• Events with valid JSON message are parsed and sent as structured data
• Non-JSON messages are sent as {"text": "message"} objects
• Supports nested JSON structures and complex data types

Application Context:

• applicationName: Required field for log categorization
• subsystemName: Required field for subsystem identification
• computerName: Optional field for source host identification

Timestamp Handling:

• Default: Current timestamp when event is sent
• use_timestamp: true: Use event's original timestamp
• use_hires_timestamp: true: Include nanosecond precision

Performance Considerations

Batch Processing:

• Events are buffered until batch_size is reached
• Flush occurs on batch limit or during finalization
• Larger batches reduce API calls but increase latency
• Maximum recommended batch size: 1000 events

Connection Pooling:

• HTTP client maintains connection pool
• Maximum 100 idle connections total
• Maximum 10 idle connections per host
• 90-second idle connection timeout

Retry Logic:

• Failed sends are retried based on sender configuration
• HTTP errors include response body for troubleshooting
• Check IBM Cloud Logs service status for API issues

Batch Size Limits

IBM Cloud Logs API has limits on batch size and request payload. Configure batch_size appropriately for your event size to avoid API rejections.

Error Handling

Authentication Failures:

• HTTP 401: Invalid or expired IAM token
• Refresh IAM token and restart DataStream
• Check token format (must include "Bearer " prefix)

API Errors:

• HTTP 400: Malformed request or invalid event structure
• HTTP 500: IBM Cloud Logs service error
• Error responses include detailed message for troubleshooting

Validation Errors:

• Invalid region codes are rejected during configuration validation
• Invalid severity levels default to default_severity
• Missing required fields (instance_id, application_name, subsystem_name) prevent target initialization

Examples

Basic Configuration

Sending logs to IBM Cloud Logs using token authentication...

targets: - name: ibm-logs type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: datastream subsystem_name: telemetry

With Secret Authentication

Using environment variable for secure IAM token storage...

targets: - name: ibm-logs-secure type: ibmcloudlogs properties: instance_id: xyz789abc123 region: eu-gb authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: security subsystem_name: audit computer_name: production-server

With Custom Severity

Setting default severity to Warning for important events...

targets: - name: ibm-logs-warnings type: ibmcloudlogs properties: instance_id: abc123def456 region: us-east iam_token: "${IBM_IAM_TOKEN}" application_name: monitoring subsystem_name: alerts default_severity: 4 use_timestamp: true

High-Volume Configuration

Optimizing for high-volume log ingestion with larger batches...

targets: - name: ibm-logs-high-volume type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: streaming subsystem_name: events batch_size: 1000 timeout: 30 use_timestamp: true use_hires_timestamp: true

Multi-Region Configuration

Sending logs to different IBM Cloud regions for geographic distribution...

targets: - name: ibm-logs-us type: ibmcloudlogs properties: instance_id: us123abc456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: us-region - name: ibm-logs-eu type: ibmcloudlogs properties: instance_id: eu456def789 region: eu-de iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: eu-region

With Normalization

Applying ECS normalization before sending to IBM Cloud Logs...

targets: - name: ibm-logs-normalized type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: security subsystem_name: normalized field_format: ECS default_severity: 3

Production Configuration

Production-ready configuration with secret authentication, batch optimization, and high-resolution timestamps...

targets: - name: ibm-logs-production type: ibmcloudlogs properties: instance_id: prod123abc456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: production-datastream subsystem_name: telemetry-processing computer_name: datastream-director-01 default_severity: 3 use_timestamp: true use_hires_timestamp: true batch_size: 1000 timeout: 30 field_format: ASIM