Azure Blob Storage
Synopsis
Azure Blob Storage device reads files from an Azure Blob Storage container. By default it dequeues messages from an Azure Storage Queue that notifies the device of new blobs. In poll mode it lists the container on a schedule instead, with no queue.
Schema
- id: <numeric>
name: <string>
description: <string>
type: azblob
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
container_name: <string>
connection_string: <string>
tenant_id: <string>
client_id: <string>
client_secret: <string>
account: <string>
blob_size: <numeric>
mode: <enum>
prefix: <string>
path_prefix: <string>
file_name_filter: <string>
timezone: <string>
workers: <numeric>
max_size_archive_bytes: <numeric>
max_files_in_archive: <numeric>
timeout: <numeric>
batch_size: <numeric>
pollers: <numeric>
visibility_timeout: <numeric>
interval: <numeric>
cron: <string>
start_date: <string>
end_date: <string>
max_retries: <numeric>
Configuration
The following fields are used to define the device:
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | Unique numeric identifier | |
name | Y | Device name | |
description | N | - | Optional description |
type | Y | Device type identifier (must be azblob) | |
tags | N | - | Array of labels for categorization |
pipelines | N | - | Array of preprocessing pipeline references |
status | N | true | Enable/disable the device |
Connection
The device requires either a connection string or a service principal with account name. container_name is required in all cases.
| Field | Required | Default | Description |
|---|---|---|---|
container_name | Y | Container to read. In queue mode it is also the name of the Azure Storage Queue that notifies the device of new blobs | |
connection_string | Y* | Azure Storage connection string for authentication | |
tenant_id | Y* | Azure tenant ID for service principal authentication | |
client_id | Y* | Azure client ID for service principal authentication | |
client_secret | Y* | Azure client secret for service principal authentication | |
account | Y* | Azure storage account name for service principal authentication | |
blob_size | N | 100000000 | Maximum blob size in bytes to process (100 MB) |
mode | N | queue | Ingestion mode: queue or poll. See Ingestion Modes |
prefix | N | - | Restrict ingestion to keys under this prefix. In poll mode it may carry date tokens |
path_prefix | N | - | Read in place of prefix when prefix is not set, which is how a device saved by an older console applies its prefix. A value the mode cannot use, date tokens or a stray % in queue mode, or a malformed template in poll mode, is ignored: the whole container is read and the Director log carries a warning, path_prefix "<value>" is ignored, so the whole container is read: <reason>; set prefix instead. The same value in prefix is rejected instead: the device fails at startup, and its connection state gives the reason |
file_name_filter | N | .* | Regular expression a blob name must match to be downloaded. An invalid expression is ignored and the previous filter is kept |
timezone | N | UTC | IANA name or offset stamped on every record. In poll mode it also applies to an offset-less or relative start_date and end_date |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1-64 |
max_size_archive_bytes | N | 104857600 | Largest total size of the files extracted from one archive, in bytes; larger archives are skipped. 0 or an empty value uses the default |
max_files_in_archive | N | 100 | Most files extracted from one archive; archives with more files are skipped. 0 or an empty value uses the default |
timeout | N | 5 | queue mode only. Idle backoff in seconds when the notification queue is empty (1-3600) |
batch_size | N | 32 | queue mode only. Maximum queue messages retrieved per dequeue (1-32; Azure caps at 32) |
pollers | N | 2 | queue mode only. Concurrent loops that dequeue notifications (1-16) |
visibility_timeout | N | 300 | queue mode only. Seconds a dequeued notification stays hidden while its blob is read and delivered (1-604800). Set it longer than that takes, or the blob is read twice |
interval | N | 60 | poll mode only. Seconds between list cycles. See Poll Mode Properties |
cron | N | - | poll mode only. Cron schedule for the list cycle. See Poll Mode Properties |
start_date | N | - | poll mode only. Lower bound of the listing window. See Poll Mode Properties |
end_date | N | - | poll mode only. Upper bound of the listing window. See Poll Mode Properties |
max_retries | N | 0 | poll mode only. Consecutive cycles a blob may fail before it is skipped. 0 means never give up |
A numeric value outside its range is ignored and the default applies.
* = Conditionally required: either connection_string or the combination of tenant_id, client_id, client_secret, and account must be provided.
Ingest Buffer
| Field | Required | Default | Description |
|---|---|---|---|
max_buffer_size | N | 128MB | Ingest buffer admission cap. Also sizes this listener's ingest buffers: 3/8 of this value at start, growing to at most 3x |
max_buffer_size is the ingest-buffer admission cap, and is distinct from any buffer_size field above, which sizes the network read buffer. Left unset it is 128MB, whose buffers take 48 MiB of heap per listener worker at start and grow to at most 384 MiB as bursts need the room — and with reuse: true a device runs one worker per listener. Size strings are binary, and 32MB and 32MiB are exact synonyms.
See Listener Memory Sizing for the arithmetic, the sizing table, and what happens when the cap is exhausted.
Details
Ingestion Modes
This device offers two ingestion modes, selected with mode:
mode | Behaviour |
|---|---|
queue | Default. The device consumes notifications from the bucket's event queue and fetches each object as it is announced. |
poll | The device lists the bucket on a schedule and ingests what it has not seen before. No queue is needed. |
Any other value fails with mode field not valid.
Poll mode suits a bucket with no notification wiring, and a historical backfill, since a listing can be bounded to a date range. It costs a list operation per cycle, so it is the wrong choice for a high-rate bucket that already has a queue.
Poll Mode Properties
These are read only when mode: poll is set.
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | 60 | Seconds between list cycles. When cron is set as well, a cycle starts when either is due. The default applies only when cron is unset |
cron | N | - | Cron expression for the list cycle. Alone, it is the only schedule; with interval, a cycle starts when either is due. An invalid expression is logged and the interval schedule is used instead |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1–64 |
max_retries | N | 0 | Consecutive cycles an object may fail transiently before the poller gives up on it and moves the cursor past it. 0 means never give up |
start_date | N | - | Lower bound of the listing window. Accepts Unix seconds, RFC 3339, a timestamp without an offset (read in timezone), or a relative expression. Unset or negative means the whole bucket |
end_date | N | - | Upper bound, exclusive. Unset means unbounded; a negative value is an error, and a value at or before start_date is rejected |
timezone | N | UTC | Zone used to interpret an offset-less start_date or end_date |
The Cursor
Poll mode keeps a since-cursor so a restart or a cluster failover does not re-ingest what it already has. The cursor is scoped to the listing target — the bucket and prefix together — so changing either starts a fresh one.
A stored cursor takes precedence over start_date, but only while the window is unchanged: editing start_date, end_date or prefix discards the cursor and re-evaluates the new window. A bounded window that has been fully drained stops listing and idles until the window changes.
Date Tokens in prefix
In poll mode prefix may carry date tokens, so a listing walks only the partitions that fall inside its window: %Y, %y, %m, %d, %j, %H, and %% for a literal percent. The finest token present sets the partition step.
A token-bearing prefix needs a bounded start_date — there is no way to walk partitions from the beginning of a bucket — and a window spanning too many partitions is rejected rather than expanded.
Date tokens are rejected in queue mode. A queue subscription filters on a fixed key prefix, which cannot be re-expanded per cycle, so a prefix containing %Y and the like fails at apply time. Write %% where a literal percent is wanted.
IAM Permissions
When using service principal authentication, the following Azure RBAC roles are required:
| Mode | Azure Role | Narrowest Scope |
|---|---|---|
poll | Storage Blob Data Reader | The container in container_name |
queue | Storage Queue Data Message Processor | The queue, which is named by container_name |
queue | Storage Blob Data Reader | Every container the Event Grid subscription sends events for, or the storage account |
These roles are enough. No role on the storage account itself is needed: neither Reader nor Storage Account Contributor, which can list the account keys.
- The queue role is for
queuemode only. Apolldevice needs no queue role and no queue. - In
queuemode the device reads the container from each event, not from its configuration. If the subscription sends events for a container the principal cannot read, each such blob fails with a 403, and its message goes back on the queue to be read again aftervisibility_timeout, over and over. Filter the subscription to the containers granted. See Event Grid Subscription. - A role assignment condition that limits the principal to certain paths works, as long as it allows the paths the device lists. The startup check never lists the container root unless
prefixis empty.
At startup the device checks its access with one request of the kind it reads with: in poll mode it lists one blob of the container, under prefix as it renders for the current time; in queue mode it peeks one message of the queue, which neither hides nor removes it. A role the device lacks fails it at startup with AuthorizationPermissionMismatch.
A failed check stops the device from starting, and its connection state gives the reason. Besides a missing role, the check catches:
pollmode, a container incontainer_namethat does not exist: a 404,ContainerNotFound.pollmode, a malformedprefixsuch aslogs/%Q/: the prefix error.
In queue mode the check does not reach the blob endpoint, so a device whose blob endpoint is unreachable, behind a firewall or a private endpoint, still starts. Each blob read then fails until the endpoint is reachable, and its message is read again after visibility_timeout.
Queue-Based Notification
The device uses an Azure Storage Queue as a notification mechanism. The queue name is specified by container_name. When a blob lands in the storage account, Azure emits a queue message pointing to the blob. The device dequeues these messages and downloads the referenced blobs for processing.
Event Grid Subscription
The queue messages come from an Event Grid subscription on the storage account, which you create. Without it, uploaded blobs send nothing and the queue stays empty. From the storage account in the Azure Portal:
- Select Events → + Event Subscription.
- Set Event Schema to Event Grid Schema.
- Enter any System Topic Name. It is asked for on the account's first subscription.
- In Filter to Event Types, select Blob Created only. The device drops other event types.
- Set Endpoint Type to Storage Queues, and choose the queue named by the device's
container_name. Leave the message encoding at its default, Base64, which the device decodes. - On the Filters tab, select Enable subject filtering, and set Subject Begins With to
/blobServices/default/containers/<container>/. This keeps events to the containers the principal can read. See IAM Permissions.
Choose Event Grid Schema, not Cloud Event Schema. The device reads eventType, eventTime and subject, and silently drops a CloudEvents message as empty.
- Blobs uploaded before the subscription existed send no events. Only new ones do.
- The storage account must be StorageV2, BlobStorage or BlockBlobStorage. General-purpose v1 accounts send no blob events.
Microsoft.EventGridmust be registered as a resource provider on the subscription.- If the account limits public network access, Event Grid cannot write to the queue on its own. Give the system topic a managed identity, grant it Storage Queue Data Message Sender on the queue, choose that identity for delivery, and allow trusted Microsoft services on the account.
- The device reads only
.json,.jsonl,.parquet(also.parqand.pq),.log,.txtand archives (.gz,.zip,.bz2,.tar,.tgz,.tbz2). It deletes the message for any other blob without reading it. - A
.jsonblob may hold one JSON object, an array, or several JSON values such as one object per line, so Azure diagnostic logs archived to storage (PT1H.json) are read record by record. A.gzblob whose content is not compressed is read as the file its name stands for, so a plainPT1H.json.gzis read asPT1H.json. - To check delivery, upload a test blob and open the event subscription's metrics: matched, delivered and failed events. Once the device runs it takes messages within seconds, so an empty queue means it is working only when the metrics show events delivered. An empty queue with no delivered events means the subscription is missing, its filters match nothing, or delivery failed.
Authentication
When connection_string is provided, the device builds both the blob and queue clients directly from the connection string. When account is provided instead, the device constructs clients using the service principal credentials and builds URLs of the form https://<account>.blob.core.windows.net/ and https://<account>.queue.core.windows.net/<container_name>/.
Replay
The azblobreplay device type is this collector permanently in poll mode. It lists the container for a bounded time window and ingests the blobs it finds, without an Event Grid storage queue.
Every property behaves as documented under Poll Mode above, including start_date, end_date, timezone, the date tokens in prefix, and the cursor. The only configuration difference is type: mode may be omitted, since a replay can only poll, and setting it to queue is rejected.
What the replay type adds is a reported run — live progress, a run history, and the pause, resume, cancel and run-now controls. See Replay Devices.
Examples
Connection String Authentication
Connecting with an Azure Storage connection string... | |
Service Principal Authentication
Connecting with service principal credentials... | |
Large Blob Processing
Increasing the blob size limit for large files... | |