IBM Event Streams

IBM Cloud Message Queue

Synopsis

The IBM Event Streams target writes log messages to IBM's managed Kafka service on IBM Cloud with full Kafka API compatibility. IBM Event Streams is IBM Cloud's managed Kafka service, and the service handles scaling and availability for you. Configuration follows Apache Kafka patterns with IBM Cloud-specific authentication.

Schema

- name: <string>
  description: <string>
  type: ibmeventstreams
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <integer>
    client_id: <string>
    topic: <string>
    algorithm: <string>
    username: <string>
    password: <string>
    compression: <string>
    compression_level: <string>
    acknowledgments: <string>
    allow_auto_topic_creation: <boolean>
    disable_idempotent_write: <boolean>
    max_bytes: <integer>
    max_events: <integer>
    batch_mode: <string>
    batch_separator: <string>
    field_format: <string>
    tls:
      status: <boolean>
      insecure_skip_verify: <boolean>
      min_tls_version: <string>
      max_tls_version: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be ibmeventstreams
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

IBM Event Streams Connection

Field	Required	Default	Description
address	Y	-	IBM Event Streams broker address (from service credentials). Supports a single address or a comma-separated list for multiple seed brokers. Each entry may include a port; if omitted, the port field value is appended.
port	N	9092	Broker port. IBM Event Streams listens on 9093 (SASL_SSL), so set it explicitly
client_id	N	-	Client identifier for connection tracking
topic	Y	-	Kafka topic name for message delivery

Authentication

Field	Required	Default	Description
algorithm	N	"none"	Authentication mechanism. IBM Event Streams requires scram-sha-512, so set it explicitly — the default does not authenticate
username	Y	-	IBM Event Streams username (from service credentials)
password	Y	-	IBM Event Streams password (from service credentials)

Producer Settings

Field	Required	Default	Description
compression	N	"none"	Message compression: none, gzip, snappy, lz4, zstd
compression_level	N	-	Compression level (algorithm-specific)
acknowledgments	N	"leader"	Acknowledgment level: none, leader, all. none and leader are silently upgraded to all unless disable_idempotent_write: true is also set — the idempotent producer is on by default and requires all
allow_auto_topic_creation	N	false	Allow automatic topic creation if topic doesn't exist
disable_idempotent_write	N	false	Turn off the idempotent producer. Required if you want acknowledgments to actually take effect as leader or none; leaving it false keeps exactly-once-per-partition write semantics

Batch Configuration

Field	Required	Default	Description
max_bytes	N	1048576	Maximum batch size in bytes (0 = unlimited, max: 104857600)
max_events	N	1000	Maximum number of events per batch
batch_mode	N	individual	Output format: individual (one message per event), json_array (array of JSON objects), jsonl (JSON Lines with separator)
batch_separator	N	,	Separator between messages when using jsonl batch mode
field_format	N	-	Data normalization format. See applicable Normalization section

TLS Configuration

Field	Required	Default	Description
tls.status	N	false	Enable TLS/SSL encryption. IBM Cloud connections require it, so set it to true explicitly
tls.insecure_skip_verify	N	false	Skip TLS certificate verification (not recommended)
tls.min_tls_version	N	tls1.2	Minimum TLS version: tls1.0, tls1.1, tls1.2, tls1.3
tls.max_tls_version	N	tls1.3	Maximum TLS version: tls1.0, tls1.1, tls1.2, tls1.3
tls.ca_name	N	-	CA bundle used to verify the broker certificate. When unset the host trust store is used; when set it replaces it
tls.cert_name	N	"cert.pem"	Client certificate file name for mTLS
tls.key_name	N	"key.pem"	Private key file name for mTLS
tls.passphrase	N	-	Passphrase for encrypted private key

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Authentication

SASL/SCRAM-SHA-512:

• IBM Event Streams uses SASL/SCRAM-SHA-512 authentication
• Set algorithm: scram-sha-512 explicitly, because the field defaults to none and sends no credential
• Username and password from IBM Cloud service credentials
• TLS encryption required for authentication

Service Credentials:

• Obtain credentials from IBM Cloud console
• Open your Event Streams instance and go to Service Credentials
• Create new credentials with appropriate permissions
• Extract kafka_brokers_sasl, user, and password values

Service Credential Format

IBM Event Streams service credentials include:

• kafka_brokers_sasl: Array of broker addresses
• user: SASL username for authentication
• password: SASL password for authentication
• Use any broker address from the array

Connection Configuration

Broker Addresses:

• IBM Event Streams provides multiple broker endpoints
• Format: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093
• Use any broker from service credentials
• Default port: 9093 (SASL_SSL)

TLS Requirements:

• TLS encryption mandatory for IBM Cloud connections
• Set tls.status: true explicitly, because the field defaults to false
• IBM Event Streams uses valid certificates
• No need for custom CA certificates

Topic Management

Topic Creation:

• Pre-create topics in IBM Cloud console
• Configure allow_auto_topic_creation: true for automatic creation (not recommended)
• Topic configuration managed through IBM Cloud UI or CLI

Topic Permissions:

• Service credentials grant topic-level permissions
• Writer role required for producing messages
• Configure permissions in IBM Cloud console

Performance Optimization

Batch Configuration:

• Larger batches improve throughput
• Balance batch size against latency requirements
• IBM Event Streams handles high-throughput workloads

Compression:

• Enable compression to reduce bandwidth costs
• Recommended: snappy (fast) or zstd (high compression)
• Compression reduces network transfer and storage

Connection Pooling:

• Maintains persistent connection to IBM Event Streams
• Automatic reconnection on connection loss
• Configurable client ID for connection tracking

IBM Cloud Pricing

IBM Event Streams charges based on throughput and storage. Enable compression and tune batch sizes to optimize costs.

Kafka API Compatibility

DataStream uses the standard Kafka Producer API with support for:

• Idempotent writes
• Batch compression
• SASL authentication
• TLS encryption

Examples

Basic Configuration

Sending logs to IBM Event Streams using SASL/SCRAM authentication...

targets: - name: ibm-event-streams type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: application-logs algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

With Compression

Enabling Snappy compression for bandwidth efficiency...

targets: - name: ibm-event-streams-compressed type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: telemetry-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: snappy tls: status: true

High-Volume Configuration

Optimizing for high-volume ingestion with larger batches and compression...

targets: - name: ibm-event-streams-high-volume type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: metrics-stream algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: zstd compression_level: "3" max_events: 1000 max_bytes: 1048576 acknowledgments: all tls: status: true

With Client Identification

Using client ID for connection tracking and monitoring...

targets: - name: ibm-event-streams-identified type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 client_id: datastream-director-01 topic: security-logs algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

Multi-Topic Publishing

Publishing different event types to separate topics...

targets: - name: ibm-es-security type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: security-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true - name: ibm-es-application type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: application-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

With Normalization

Applying ECS normalization before sending to IBM Event Streams...

targets: - name: ibm-event-streams-normalized type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: normalized-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" field_format: ECS compression: zstd tls: status: true

Production Configuration

Production-ready IBM Event Streams configuration with compression, batching, TLS, and full acknowledgments...

targets: - name: ibm-event-streams-production type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 client_id: datastream-production-01 topic: production-telemetry algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: zstd compression_level: "3" acknowledgments: all max_events: 1000 max_bytes: 1048576 field_format: ASIM tls: status: true min_tls_version: tls1.2

Troubleshooting

The ibmeventstreams target connects, authenticates and produces in exactly the same way as the kafka target, so it reports the same errors for the same causes: authentication and encryption failures at startup, topic authorization and missing topic failures on the first batch, and the size limits on a single record and on a whole batch. Each error is listed with its cause and its fix on Apache Kafka. Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name in the examples there. See Target Delivery Errors for how Director logs and retries target failures.

What differs for IBM Event Streams

Three settings have to be written out before the connection can work. None of them is preset for this service.

• algorithm starts at none and has to be set to scram-sha-512. This is the mechanism the service offers, and the listener typically accepts that one only. A target left at the default sends no credential at all and fails at startup with is SASL missing?. A target set to scram-sha-256 or to plain is then refused with UNSUPPORTED_SASL_MECHANISM before the credential is read, so a correct user name and password do not help. Write the value with hyphens, because scram_sha_512 is rejected as an invalid algorithm.

• tls.status starts at false and has to be set to true. The service listens with TLS only. Without it the connection is closed during negotiation and the reason ends in is TLS missing?. The service presents a certificate from a public authority, so tls.ca_name is typically not needed and the host trust store is enough.

• port starts at 9092 and has to be set to the port in your service credentials, which is 9093 for the SASL and TLS listener. Left at the default, the target reports a connection failure against a port that is not open, typically i/o timeout or a refused connection.

Reading the values out of the service credentials

Create a service credential in the IBM Cloud console and read the three connection values out of it. The bootstrap host list becomes address, as one entry or as a comma-separated list. The user name and the API key become username and password.

The broker entries usually include :9093, and an entry that carries its own port keeps it. Only entries without one have port appended, so a list that mixes the two forms is fine.

note

Producing to a topic needs the writer role on that topic, granted to the service credential in the IBM Cloud console. The role is not tested at startup, so a target with working credentials and no writer role reports connected and then fails every batch with TOPIC_AUTHORIZATION_FAILED.