Azure Service Bus

Microsoft Azure Message Queue

Synopsis

Creates a receiver that runs on the Director and receives from one Azure Service Bus entity: a queue, or one subscription of a topic, or that entity's dead-letter or transfer dead-letter sub-queue. A message is completed only once its delivery to the target is confirmed (see Settlement). Session-enabled entities are supported.

The device receives from an entity that already exists. It does not create queues, topics, subscriptions or rules, it reads one entity per device, and it does not carry message properties or application properties into the records. To send to Service Bus, use the Azure Service Bus target, whose sign-in fields this device shares.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: servicebus
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    client_connection_string: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    namespace: <string>
    transport: <string>
    queue: <string>
    topic: <string>
    subscription: <string>
    sub_queue: <string>
    receive_mode: <string>
    sessions: <boolean>
    max_sessions: <numeric>
    workers: <numeric>
    max_messages: <numeric>
    prefetch_count: <numeric>
    max_lock_renewal: <numeric | string>
    content_encoding: <string>
    max_decompression_size: <numeric | string>
    event_split_rules: <string[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be servicebus
tags	N	-	Optional tags
pipelines	N	-	Optional pre-processor pipelines
status	N	true	Enable/disable the device

Sign-In

The sign-in is chosen by which fields are set, and the first match wins. These are the same fields, read in the same order, as the Azure Service Bus target's.

Fields set	Sign-in	Needs namespace
client_connection_string	Shared access key, namespace-level or entity-level	N
tenant_id, client_id, and client_secret	Microsoft Entra ID service principal	Y
client_id only	User-assigned managed identity	Y
None of them	Default credential chain (DefaultAzureCredential), tried in order: environment variables, workload identity, managed identity, then developer sign-ins such as the Azure CLI. Grant the receive right to the identity the chain reaches first	Y
Any other mix, such as tenant_id and client_id without client_secret	Refused as a half-written service principal	-

Field	Required	Default	Description
client_connection_string	N	-	Service Bus connection string. When set, the other sign-in fields are not used
tenant_id	N	-	Azure tenant ID of the service principal
client_id	N	-	Client ID of the service principal, or, set alone, of a user-assigned managed identity
client_secret	N	-	Client secret of the service principal
namespace	Y*	-	The namespace: a bare name, such as contoso, which becomes contoso.servicebus.windows.net, or a full host name, which is used as given
transport	N	amqp	amqp connects with AMQP over TLS on TCP port 5671. amqp_websockets connects with AMQP over WebSockets on TCP port 443, for networks that only allow HTTPS out

* = Required for every sign-in except a connection string

Entity

Set exactly one entity: queue, or topic together with subscription.

Field	Required	Default	Description
queue	Y*	-	The queue to receive from
topic	Y*	-	The topic whose subscription to receive from. Requires subscription
subscription	Y*	-	The subscription of topic to receive from. Requires topic
sub_queue	N	none	none receives from the entity itself. deadletter drains the entity's dead-letter queue, and transferdeadletter its transfer dead-letter queue

* = Either queue, or topic and subscription, but not both

Receiving

Field	Required	Default	Description
receive_mode	N	peeklock	peeklock locks each message until the device settles it. receiveanddelete has Service Bus delete each message as it is received. See Settlement
sessions	N	false	Receive from a session-enabled entity, one session at a time per worker. See Sessions
max_sessions	N	8	Sessions received at once, 1–100. Read only with sessions: true
workers	N	4	Concurrent receivers, 1–64. Not read with sessions: true, where max_sessions sets the number of receivers
max_messages	N	100	Messages asked for per receive, 1–5000
prefetch_count	N	0	Messages a worker holds received ahead of the pipeline, 0–5000. 0 holds none ahead
max_lock_renewal	N	5m	How long a held message's lock is renewed, from 0 (off) to 1h. A number of seconds, or a duration such as 10m. See Lock Renewal

Decoding

Field	Required	Default	Description
content_encoding	N	identity	How a message body is read: identity (as it came), gzip, or auto (follows the message's own content encoding), as on the RabbitMQ device
max_decompression_size	N	64MB	The most a body is decompressed to, 1KB–1GB

Event Split Rules

Field	Required	Default	Description
event_split_rules	N	-	Ordered list of rule set keys that split each message into separate logs. See Event Split Rules

Without rules, the device keeps each message whole, as one record (see Records), unless its split_mode splits it. With them, a message that holds many logs, such as a JSON array or several lines, is split as the device receives it, and every later count, the device's own included, counts the logs. Under peeklock, a message is still completed only after all of its logs are delivered. Under receiveanddelete, Service Bus deletes it as it is received, as without rules (see Settlement). A condition reads the entity as source, as the record's request names it: the queue, or <topic>/<subscription>. Changing the list restarts the device; editing a rule set it uses does not.

Validation

Every value outside its bounds is refused, and the device's state is Configuration error, naming the field. Nothing is silently adjusted. These combinations are refused as well:

• both queue and topic, or topic without subscription, or subscription without topic
• sessions: true together with a sub_queue other than none, because a dead-letter queue is not session-enabled. Set sessions: false to drain it

The device starts, and logs a warning, when a setting has no effect or puts messages at risk:

• receive_mode: receiveanddelete, which loses a message the pipeline fails to deliver
• max_lock_renewal set under receiveanddelete, where no message is locked
• workers set with sessions: true, or max_sessions set without it
• prefetch_count above 0 under peeklock with max_lock_renewal: 0, where a message held longer than its lock is sent again

Details

IAM Permissions

Sign-in	Required right	Scope
Service principal or managed identity	Azure Service Bus Data Receiver	The namespace, or the queue, topic or subscription
Connection string	A shared access policy carrying the Listen claim	The namespace, or the entity the string is scoped to with EntityPath=

Settlement

Under peeklock, Service Bus locks each message to the receiver that took it until the device settles it:

Mode and event	What happens to the message
peeklock, delivered to the target	Completed once the target confirmed its delivery
peeklock, the device stops first	Abandoned after a 10-second grace, and Service Bus sends it again
peeklock, sub_queue: none, the body cannot be decoded, or is larger than the device can ever buffer	Dead-lettered, with the reason virtualmetric and the error as its description
peeklock, sub_queue: deadletter or transferdeadletter, the body cannot be decoded, or is larger than the device can ever buffer	Completed, so it is deleted, and the error is logged. A message already in a dead-letter queue cannot be dead-lettered again
peeklock, held longer than max_lock_renewal	Its lock is no longer renewed and lapses, and Service Bus sends it again. This is logged once per message
receiveanddelete	Deleted by Service Bus as it is received. A pipeline or target failure loses the message

Delivery under peeklock is at-least-once, so a message can be written twice. A message with an empty body is completed and writes no record.

warning

receive_mode: receiveanddelete trades delivery for throughput. A message the pipeline or the target fails to deliver is gone, and a message that cannot be decoded is dropped rather than dead-lettered. The device logs a warning at start.

Lock Renewal

A peek-locked message's lock lasts as long as the entity's lock duration. The device renews the lock of every message it holds, until the message is settled or until max_lock_renewal has passed since it was received. The default of 5 minutes covers normal batching.

With max_lock_renewal: 0, no message lock is renewed. A message whose delivery takes longer than the entity's lock duration is then sent again by Service Bus, which writes it twice.

Sessions

With sessions: true, the device receives from a session-enabled queue or subscription. max_sessions workers each hold one session at a time and renew the session's lock while they hold it. A worker gives a session up once the session is idle and none of its messages is still outstanding, and then takes the next session that has messages.

Records

Each message is one record, its body decoded as content_encoding says, unless event_split_rules or the device's split_mode splits it into several (see Event Split Rules). The record's request is the entity: <queue> or <topic>/<subscription>, with /$deadletterqueue appended when the device drains either sub-queue. The record time is the message's enqueued time.

States

State	When
Connected	Every worker receives
Connection error	A receive failed for a transient reason, such as a busy namespace or a lost connection, or nothing came through a receiver's one-minute wait and the namespace did not answer the 20-second check after it. It is retried with a backoff from 1 to 30 seconds
Authentication error	The sign-in was refused, or the identity may not receive from the entity (401, 403). The state's message names the role and claim to check
Configuration error	The configuration is refused (see Validation), or the namespace has no such queue, topic or subscription

An authentication or configuration error stops the device, and the Director starts it again after a growing wait. Any configuration change restarts the device.

Clusters of Directors

Every Director of a cluster may run the device. Its receivers are competing consumers: Service Bus locks each message, or each session, to one receiver, so the nodes share the entity's messages.

Examples

Queue with Connection String

Receiving from a queue with a connection string...

- id: 1 name: orders_servicebus type: servicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "orders"

Topic Subscription with Service Principal

Receiving from one subscription of a topic, signed in as a service principal...

- id: 2 name: audit_servicebus type: servicebus properties: tenant_id: "12345678-1234-1234-1234-123456789012" client_id: "87654321-4321-4321-4321-210987654321" client_secret: "${AZURE_CLIENT_SECRET}" namespace: "contoso" topic: "audit-events" subscription: "datastream"

Managed Identity over WebSockets

Signing in as a user-assigned managed identity, through a network that only allows HTTPS out...

- id: 3 name: mi_servicebus type: servicebus properties: client_id: "87654321-4321-4321-4321-210987654321" namespace: "contoso.servicebus.windows.net" transport: "amqp_websockets" queue: "security-logs"

Draining a Dead-Letter Queue

Receiving what Service Bus has dead-lettered on a queue...

- id: 4 name: orders_dlq_servicebus type: servicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "orders" sub_queue: "deadletter"

note

A message this device cannot decode is deleted, not kept (see Settlement).

Session-Enabled Queue

Receiving from a session-enabled queue, sixteen sessions at once...

- id: 5 name: sessions_servicebus type: servicebus properties: namespace: "contoso" queue: "ordered-transactions" sessions: true max_sessions: 16

High-Volume

Raising the receivers and the batch size for a busy queue, with a longer lock renewal for slow targets...

- id: 6 name: high_volume_servicebus type: servicebus properties: client_connection_string: "${SERVICEBUS_CONNECTION_STRING}" queue: "high-volume-logs" workers: 16 max_messages: 500 prefetch_count: 500 max_lock_renewal: "15m" content_encoding: "auto"