WarpStream

Kafka-Compatible Message Queue

Send processed telemetry data to WarpStream serverless Kafka-compatible service.

Synopsis

The WarpStream target writes log messages to WarpStream's serverless Kafka-compatible platform with object storage backend. WarpStream provides Kafka API compatibility with zero-disk architecture, automatic scaling, and consumption-based pricing. Configuration follows Apache Kafka patterns with WarpStream-specific endpoints and authentication.

Schema

targets:
  - name: <string>
    type: warpstream
    properties:
      address: <string>
      port: <integer>
      client_id: <string>
      topic: <string>
      algorithm: <string>
      username: <string>
      password: <string>
      compression: <string>
      compression_level: <string>
      acknowledgments: <string>
      allow_auto_topic_creation: <boolean>
      disable_idempotent_write: <boolean>
      max_bytes: <integer>
      max_events: <integer>
      tls:
        status: <boolean>
        insecure_skip_verify: <boolean>
        min_tls_version: <string>
        max_tls_version: <string>
        cert_name: <string>
        key_name: <string>
        passphrase: <string>
      field_format: <string>
      interval: <string|numeric>
      cron: <string>

Configuration

Base Target Fields

Field	Type	Required	Description
name	string	Y	Unique identifier for this target
description	string	N	Human-readable description
type	string	Y	Must be warpstream
pipelines	array	N	Pipeline names to apply before sending
status	boolean	N	Enable (true) or disable (false) this target

WarpStream Connection

Field	Type	Required	Description
address	string	Y	WarpStream broker address (provided in console)
port	integer	N	WarpStream broker port. Default: 9092
client_id	string	N	Client identifier for connection tracking
topic	string	Y	Kafka topic name for message delivery

Authentication

Field	Type	Required	Description
algorithm	string	N	Authentication mechanism. Default: plain
username	string	Y	WarpStream API key
password	string	Y	WarpStream API secret

Producer Settings

Field	Type	Required	Description
compression	string	N	Message compression (none, gzip, snappy, lz4, zstd). Default: none
compression_level	string	N	Compression level (algorithm-specific)
acknowledgments	string	N	Acknowledgment level (none, leader, all). Default: leader
allow_auto_topic_creation	boolean	N	Allow automatic topic creation. Default: false
disable_idempotent_write	boolean	N	Disable idempotent producer. Default: false

Batch Configuration

Field	Type	Required	Description
max_bytes	integer	N	Maximum batch size in bytes (0 = unlimited). Default: 0
max_events	integer	N	Maximum number of events per batch. Default: 1000

TLS Configuration

Field	Type	Required	Description
tls.status	boolean	N	Enable TLS encryption. Default: false
tls.insecure_skip_verify	boolean	N	Skip TLS certificate verification. Default: false
tls.min_tls_version	string	N	Minimum TLS version. Default: tls1.2
tls.max_tls_version	string	N	Maximum TLS version. Default: tls1.3
tls.cert_name	string	N	Client certificate filename (PEM format)
tls.key_name	string	N	Client private key filename (PEM format)
tls.passphrase	string	N	Private key passphrase if encrypted

Normalization

Field	Type	Required	Description
field_format	string	N	Apply format normalization (ECS, ASIM, UDM)

Scheduler

Field	Type	Required	Description
interval	string/numeric	N	Execution frequency (realtime by default)
cron	string	N	Cron expression for scheduled execution

Details

Authentication

API Key Authentication:

• Obtain API key and secret from WarpStream console
• Use username field for API key
• Use password field for API secret
• Authentication required for all WarpStream connections

SASL/PLAIN:

• WarpStream uses SASL/PLAIN authentication mechanism
• Set algorithm: plain (default for WarpStream)
• Credentials transmitted over TLS-encrypted connection

WarpStream Credentials

Generate API keys in WarpStream console. Each key has specific permissions for topics. Ensure API key has write permissions for configured topics.

Connection Endpoints

Broker Addresses:

• WarpStream provides broker addresses in console
• Format: <cluster-name>.warpstream.com
• Default port: 9092 (Kafka protocol)
• Use provided endpoint exactly as shown in console

Multi-Region Deployment:

• WarpStream supports multi-region clusters
• Connect to regional endpoints for optimal latency
• Cross-region replication handled automatically

Topic Management

Topic Creation:

• Configure allow_auto_topic_creation: true for automatic topic creation
• Pre-create topics in WarpStream console for production use
• Topic configuration managed through WarpStream console

Topic Permissions:

• API keys grant topic-level permissions
• Verify API key has write access to configured topics
• Permission errors result in publish failures

Performance Optimization

Batch Configuration:

• Larger batches improve throughput and reduce costs
• Balance batch size against latency requirements
• WarpStream optimizes object storage writes internally

Compression:

• Enable compression to reduce bandwidth and storage costs
• Recommended algorithms: zstd (best compression), snappy (fast)
• Compression reduces WarpStream consumption-based charges

Connection Pooling:

• Maintains persistent connection to WarpStream brokers
• Automatic reconnection on connection loss
• Configurable client ID for connection tracking

Cost Optimization

WarpStream charges based on data volume processed. Enable compression and tune batch sizes to optimize costs while meeting latency requirements.

Kafka API Compatibility

Supported Features:

• Kafka Producer API
• Idempotent writes
• Batch compression
• SASL authentication
• TLS encryption

Differences from Apache Kafka:

• No ZooKeeper dependency
• Object storage backend instead of local disks
• Serverless scaling without brokers
• Different performance characteristics

Security Best Practices

Credential Management:

• Store API keys in environment variables
• Rotate API keys periodically
• Use separate API keys for different environments

TLS Encryption:

• Enable TLS for production deployments
• WarpStream enforces TLS for authentication
• Client certificates optional for mutual TLS

Examples

Basic Configuration

Sending logs to WarpStream using API key authentication...

targets: - name: warpstream-logs type: warpstream properties: address: my-cluster.warpstream.com topic: application-logs username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}"

With Compression

Enabling Zstd compression for optimal cost and bandwidth efficiency...

targets: - name: warpstream-compressed type: warpstream properties: address: my-cluster.warpstream.com topic: telemetry-events username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" compression: zstd compression_level: "3"

High-Volume Configuration

Optimizing for high-volume ingestion with larger batches and compression...

targets: - name: warpstream-high-volume type: warpstream properties: address: my-cluster.warpstream.com topic: metrics-stream username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" compression: snappy max_events: 1000 max_bytes: 1048576 acknowledgments: all

With Auto Topic Creation

Allowing automatic topic creation for dynamic topic names...

targets: - name: warpstream-auto-topic type: warpstream properties: address: my-cluster.warpstream.com topic: dynamic-logs username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" allow_auto_topic_creation: true

With TLS Encryption

Enabling TLS encryption for secure data transmission...

targets: - name: warpstream-secure type: warpstream properties: address: my-cluster.warpstream.com topic: security-logs username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" tls: status: true min_tls_version: tls1.2 max_tls_version: tls1.3

With Normalization

Applying ECS normalization before sending to WarpStream...

targets: - name: warpstream-normalized type: warpstream properties: address: my-cluster.warpstream.com topic: normalized-events username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" field_format: ECS compression: zstd

Production Configuration

Production-ready configuration with compression, batching, TLS, and acknowledgments...

targets: - name: warpstream-production type: warpstream properties: address: production-cluster.warpstream.com port: 9092 client_id: datastream-director-01 topic: production-telemetry username: "${WARPSTREAM_API_KEY}" password: "${WARPSTREAM_API_SECRET}" compression: zstd compression_level: "3" acknowledgments: all max_events: 1000 max_bytes: 1048576 field_format: ASIM tls: status: true min_tls_version: tls1.2