Databricks (Azure Blob Storage)

Data Warehouse Target

Synopsis

The Databricks Azure Blob target stages telemetry files to Azure Blob Storage, then executes COPY INTO commands on Databricks SQL warehouses to load data into Unity Catalog tables.

Schema

- name: <string>
  description: <string>
  type: azdatabricks
  properties:
    server_hostname: <string>
    http_path: <string>
    access_token: <string>
    catalog: <string>
    namespace: <string>
    account: <string>
    staging_container: <string>
    staging_prefix: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    table: <string>
    schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    tables:
      - table: <string>
        schema: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
    batch_size: <integer>
    max_size: <integer>
    timeout: <integer>
    drop_unknown_stream_events: <boolean>
    field_format: <string>
    max_rows_per_rowgroup: <numeric>
    buffer_size: <numeric>
    data_page_version: <string>
    metadata: <key-value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be azdatabricks
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Databricks Connection

Field	Required	Default	Description
server_hostname	Y	-	Databricks workspace URL (e.g., abc123.azuredatabricks.net)
http_path	Y	-	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123def456)
access_token	Y	-	Databricks personal access token
catalog	Y	-	Unity Catalog name
namespace	N	default	Databricks schema name

Azure Blob Staging Configuration

Field	Required	Default	Description
account	Y	-	Azure storage account name
staging_container	Y	-	Azure Blob container name for staging files
staging_prefix	N	databricks-staging/	Blob prefix path
tenant_id	Y	-	Azure AD tenant ID
client_id	Y	-	Service principal client ID
client_secret	Y	-	Service principal client secret

Table Configuration

Field	Required	Default	Description
table	Y*	-	Catch-all table name for all events
schema	Y*	-	Avro or Parquet schema: a Library name, a built-in name, or inline JSON. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File naming template
format	N	json	File format. Use json, avro, or parquet — see the warning below.
compression	N	zstd	Compression algorithm
extension	N	-	File extension override. Defaults to the resolved format.
tables	N	-	Multiple table configurations (see below)
tables.table	Y	-	Target table name
tables.schema	Y*	-	Avro or Parquet schema for this table: a Library name, a built-in name, or inline JSON. Parquet also accepts a deployed schema file. Avro has no file lookup
tables.name	Y	-	File naming template for this table
tables.format	N	-	File format for this table. Falls back to the catch-all format.
tables.compression	N	-	Compression algorithm for this table. Falls back to the catch-all compression.
tables.extension	N	-	File extension override for this table. Falls back to the catch-all extension.

* = At least one of table (catch-all) or tables (multiple) must be configured. For Avro/Parquet formats, schema is required.

warning

Only json, avro, and parquet are written correctly. csv, orc, and text pass configuration validation, but the staged file is written as JSON while the COPY INTO command still declares the requested format, so the load fails.

Parquet Options

These apply only when format is parquet.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Maximum rows per Parquet row group
buffer_size	N	262144	Parquet page buffer size in bytes (256KB)
data_page_version	N	V2	Parquet data page version (V1, V2)
metadata	N	-	Key/value pairs written into the Parquet file metadata

Batch Configuration

Field	Required	Default	Description
batch_size	N	100000	Maximum events per file before flush
max_size	N	33554432	Maximum file size in bytes before flush (32MB)
timeout	N	300	Timeout in seconds. It bounds the staging upload and the COPY INTO command
drop_unknown_stream_events	N	true	Drop events that do not match any configured table

Normalization

Field	Required	Default	Description
field_format	N	-	Apply format normalization (ECS, ASIM, UDM)

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Databricks

Details

Architecture Overview

The Databricks Azure Blob target implements a two-stage loading pattern:

1. Stage Files to Azure Blob: Events are written to files in Azure Blob Storage using the configured format
2. Execute COPY INTO: SQL commands load data from Blob Storage into Databricks Unity Catalog tables using ABFSS paths

Unity Catalog Integration

Catalog Structure:

• Tables are organized within Unity Catalog using three-level namespace: catalog.namespace.table
• The catalog field specifies the Unity Catalog name
• The namespace field specifies the schema (defaults to default)
• Table names are validated to ensure they are valid SQL identifiers

Warehouse ID Extraction:

• The target automatically extracts the warehouse ID from the http_path
• Example: /sql/1.0/warehouses/abc123def456 → abc123def456
• This warehouse ID is used for all COPY INTO operations

Unity Catalog Permissions

The Databricks access token requires permissions to:

• Execute SQL statements on the specified warehouse
• Write data to the target catalog and schema
• Access the Azure Blob staging location (configured separately in Databricks)

Azure Blob Staging Operations

File Upload:

• Files are staged to https://{account}.blob.core.windows.net/{container}/{prefix}/{table}/{filename} structure
• Uses Azure SDK for secure uploads with service principal authentication
• Supports Azure AD authentication through client credentials

ABFSS Path Construction:

• The target automatically constructs ABFSS paths for COPY INTO commands
• Format: abfss://{container}@{account}.dfs.core.windows.net/{prefix}/{table}/{filename}
• ABFSS protocol is used for direct Databricks access to Azure Data Lake Storage Gen2

Cleanup:

• Staged files are deleted after the COPY INTO command reaches a final state, whether it loaded the file or refused it
• A file is kept only when the outcome of the command is unknown, for example after a timeout. See the Troubleshooting section

Service Principal Authentication

Azure AD Integration:

• Uses service principal (client credentials) for Azure Blob Storage authentication
• Requires tenant_id, client_id, and client_secret configuration
• Service principal must have Storage Blob Data Contributor role on the container

Required Permissions:

• Storage Blob Data Contributor: Write and delete blobs in staging container
• Storage Blob Data Reader: Optional, for Databricks direct access

Service Principal Permissions

Ensure the service principal has appropriate permissions on both the staging container (for DataStream uploads) and the Databricks workspace (for COPY INTO access).

File Format Support

Valid Formats:

• json: Newline-delimited JSON objects
• avro: Schema-based binary format (requires schema)
• parquet: Columnar storage format (requires schema)

Write the value in lower case. csv, orc and text pass configuration validation but are not written correctly, and a capitalized value such as PARQUET is staged as JSON whatever the COPY INTO command declares.

Schema Requirements:

• Avro and Parquet formats require schema field with valid schema definition
• Schema must match the expected table structure in Databricks
• Other formats use schema inference from data

Multi-Table Routing

Catch-All Table:

• Use table field to send all events to a single table
• Simplest configuration for single-destination scenarios

Multiple Tables:

• Use tables array to route different event types to different tables
• Each table entry specifies table, schema, name, format fields
• Events routed based on SystemS3 field in pipeline

Example Configuration:

tables:
  - table: security_events
    schema: security_schema
    name: security.{{.Timestamp}}.parquet
    format: parquet
  - table: access_logs
    schema: access_schema
    name: access.{{.Timestamp}}.parquet
    format: parquet

Performance Considerations

Batch Processing:

• Events are buffered until batch_size or max_size limits are reached
• Larger batches reduce Blob API calls and COPY INTO operations
• Balance batch size against latency requirements

Upload Optimization:

• Azure SDK automatically handles large blob uploads
• Uses block blobs for efficient data transfer
• Connection pooling optimizes network performance

COPY INTO Performance:

• COPY INTO commands are executed with configurable timeout
• Failed COPY operations return errors for retry logic
• Warehouse must be running for COPY INTO to succeed

Warehouse State

Ensure the SQL warehouse is running before sending data. COPY INTO commands will fail if the warehouse is stopped. Configure warehouse auto-start or manual start procedures.

Error Handling

Upload Failures:

• Failed Blob uploads are retried based on sender configuration
• Permanent failures prevent COPY INTO execution
• Check service principal permissions and network connectivity

COPY INTO Failures:

• Schema mismatches between files and tables cause failures
• Invalid SQL identifiers (catalog, schema, table names) are rejected at validation
• Check Databricks query history for detailed error messages

Examples

Basic Configuration

Sending telemetry to Databricks using Azure Blob staging with Parquet format...

targets: - name: databricks-warehouse type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: production_data namespace: telemetry account: datastreamstaging staging_container: databricks-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: events schema: event_schema name: events.{{.Timestamp}}.parquet format: parquet

With Custom Staging Prefix

Using custom blob prefix for organized staging file structure...

targets: - name: databricks-organized type: azdatabricks properties: server_hostname: xyz789.azuredatabricks.net http_path: /sql/1.0/warehouses/xyz789def123 access_token: "${DATABRICKS_TOKEN}" catalog: security_analytics namespace: logs account: securitystorage staging_container: staging staging_prefix: datastream/databricks/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: security_events schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet

Multi-Table Configuration

Routing different event types to separate Databricks tables...

targets: - name: databricks-multi-table type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: analytics namespace: production account: analyticsstorage staging_container: staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" tables: - table: authentication_events schema: auth_schema name: auth.{{.Timestamp}}.parquet format: parquet - table: network_events schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet - table: application_logs schema: app_schema name: app.{{.Timestamp}}.parquet format: parquet

High-Volume Configuration

Optimizing for high-volume ingestion with batch limits and compression...

targets: - name: databricks-high-volume type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: high_volume_data namespace: streaming account: streamingstorage staging_container: high-volume-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 100000 max_size: 134217728 timeout: 600 table: streaming_events schema: streaming_schema name: stream.{{.Timestamp}}.parquet format: parquet compression: snappy

JSON Format

Using JSON format for flexible schema evolution and debugging...

targets: - name: databricks-json type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: development namespace: test_data account: devstorage staging_container: dev-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: test_events name: test.{{.Timestamp}}.json format: json

With Normalization

Applying ASIM normalization before loading to Databricks...

targets: - name: databricks-normalized type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: security_data namespace: normalized account: securitystorage staging_container: security-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" field_format: ASIM table: asim_events schema: asim_schema name: asim.{{.Timestamp}}.parquet format: parquet

Production Configuration

Production-ready configuration with performance tuning and multi-table routing...

targets: - name: databricks-production type: azdatabricks properties: server_hostname: production.azuredatabricks.net http_path: /sql/1.0/warehouses/prod123abc456 access_token: "${DATABRICKS_TOKEN}" catalog: production_analytics namespace: telemetry account: productionstorage staging_container: production-staging staging_prefix: datastream/databricks/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 50000 max_size: 67108864 timeout: 300 field_format: ECS tables: - table: security_events schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet compression: snappy - table: audit_logs schema: audit_schema name: audit.{{.Timestamp}}.parquet format: parquet compression: snappy - table: network_flows schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet compression: snappy

Troubleshooting

This section covers the errors you are most likely to see with the azdatabricks target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.
• The SQL warehouse's Query History in the Databricks workspace. It holds the full message of every COPY INTO statement Director submits.

See Target Delivery Errors for how Director logs and retries target failures.

Two identities are involved and they fail in different places. The Azure service principal in tenant_id, client_id and client_secret writes the staging file into the container. The Databricks access token in access_token runs the COPY INTO statement that loads that file, and the warehouse then reads the file itself over an abfss:// path. A right granted to one of the two is never granted to the other. This target requires all three service principal fields and has no Managed Identity fallback, unlike the other Azure targets.

Only the access token is exercised at startup, by a SELECT 1 against the warehouse. The Azure credential and the storage role are first used on the first upload, so a wrong secret or a missing role is reported at the first flush while the connection status still shows the target as connected.

Which permission is missing?

Match the error you see against this table first. The identity that needs the right depends on which half of the target failed.

Error text	Missing right	Where it is granted
databricks API error (status 403): {"error_code":"PERMISSION_DENIED","message":"invalid access token"}	A valid personal access token whose user has CAN USE on the SQL warehouse	Databricks workspace: the warehouse's Permissions tab, and Settings > Developer > Access tokens for the token itself
statement failed with state: FAILED, with PERMISSION_DENIED in the query history (typical)	USE CATALOG, USE SCHEMA and MODIFY for the token's user	Unity Catalog: the catalog in catalog, the schema in namespace, and each table
statement failed with state: FAILED, with the staging path named in the query history (typical)	READ FILES for the token's user	Unity Catalog: an external location and its storage credential covering abfss://mycontainer@mystorageaccount.dfs.core.windows.net/
RESPONSE 403 ... ERROR CODE: AuthorizationPermissionMismatch	Storage Blob Data Contributor	The storage account in account, or only the container in staging_container. Owner and Contributor do not carry blob write permission
RESPONSE 404 ... ERROR CODE: ContainerNotFound	None. The container must already exist	The storage account, under Data storage > Containers
RESPONSE 403 ... ERROR CODE: AuthorizationFailure	None. The storage network rules typically reject the Director host	The storage account's Networking settings
ClientSecretCredential authentication failed ... AADSTS...	None. The credential itself is wrong	The Entra app registration named by client_id

The split runs through the whole page. The first three rows are granted in the Databricks workspace, to the user behind access_token. The next three are granted in Azure, to the application in client_id. Granting one side generously does nothing for the other. The warehouse also needs its own network path to the storage account, because it reads the staged file directly rather than through Director.

note

Azure role assignments typically take several minutes to propagate. Unity Catalog grants apply to the next statement. Director retries automatically, so no restart is needed once the right is in place.

"databricks API error (status 403)" with PERMISSION_DENIED

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 3). Reason: failed to test databricks connection: databricks API error (status 403): {"error_code":"PERMISSION_DENIED","message":"invalid access token"}

Cause: at startup Director runs SELECT 1 on the warehouse named in http_path. The workspace answered and refused the token. access_token is wrong, expired or revoked, or its user has no CAN USE on that warehouse. This is the first thing to fail on a new target, because it is the first call the target makes. Other statuses arrive in the same message shape:

Status	Meaning
401, 403	The token is wrong, or its user lacks CAN USE on the warehouse
404	server_hostname is not the workspace that holds this warehouse, or the warehouse in http_path no longer exists
429 or a 5xx, with {"error_code":"REQUEST_LIMIT_EXCEEDED"} or {"error_code":"INTERNAL_ERROR"}	The workspace is throttling or has a problem of its own. Both clear without a change here
400 with INVALID_PARAMETER_VALUE	The request was refused. Check http_path, catalog and namespace. Setting timeout: 50 typically helps, because the API caps the wait it accepts

Fix: generate a new personal access token in the workspace, grant its user CAN USE on the warehouse, and set server_hostname to the bare host, my-workspace.azuredatabricks.net. An https:// prefix is not removed, so it ends up inside the request URL.

Data impact: none. The target never starts, so nothing is staged and nothing is loaded. Incoming data waits in the Director queue and is delivered once the target starts. Retries back off from 5 seconds to once a minute and continue for as long as the cause persists, so a high attempt count is not a separate fault.

"ClientSecretCredential authentication failed" with an AADSTS code

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 0 execute() failed for databricks-warehouse: failed to upload to azure blob: ClientSecretCredential authentication failed. POST https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/token
RESPONSE 401: 401 Unauthorized
{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. ..."}
To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret

Cause: the service principal credentials are wrong. The Entra token is requested on the first upload rather than at startup, so this appears at the first flush and the connection status shows the target as connected until then. The AADSTS code tells you which field:

Code	Meaning	Check
AADSTS7000215	Invalid client secret	The secret ID was pasted instead of the secret value, or the secret was rotated
AADSTS7000222	Client secret expired	Create a new secret on the app registration and update client_secret
AADSTS700016	Application not found in the directory	client_id is wrong, or tenant_id is a different tenant than the one the app registration lives in
AADSTS90002	Tenant not found	tenant_id is not a valid tenant ID

Fix: correct the field and save the target. For AADSTS7000215, paste the secret value from the app registration, not the secret ID shown next to it. Nothing is lost: the batch stays in the queue and is redelivered once the token request succeeds.

"client_secret is required for azdatabricks target", or "invalid tenantID"

[Error] [director] [target-<target id>] [databricks-warehouse] ValidateConfig failed for target "databricks-warehouse": client_secret is required for azdatabricks target
[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 1). Reason: invalid tenantID. You can locate your tenantID by following the instructions listed here: https://learn.microsoft.com/partner-center/find-ids-and-domain-names

Cause: the first line means one of tenant_id, client_id and client_secret is empty. All three are required here, so a half-filled service principal is rejected outright rather than falling back to the host's Managed Identity as the other Azure targets do. The second line means the field was filled in with something Entra does not accept as a tenant. The usual trigger is a ${AZURE_TENANT_ID} reference whose variable is not exported to the Director service user. An unset variable is passed on as the literal text ${AZURE_TENANT_ID}, which is not empty, so it clears the required-field check and fails here instead. A workspace URL pasted in place of the tenant GUID ends the same way.

Fix: fill in all three fields. Export the variables for the account that runs the Director service, not only in your own shell, or paste the tenant GUID such as 00000000-0000-0000-0000-000000000000 directly.

Data impact: none. Nothing is staged while either message repeats, and queued data is delivered once the target starts. The configuration is re-checked about every 30 seconds.

"RESPONSE 403" with ERROR CODE AuthorizationPermissionMismatch

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 2 execute() failed for databricks-warehouse: failed to upload to azure blob: PUT https://mystorageaccount.blob.core.windows.net/mycontainer/databricks-staging/my_table/vmetric.1757590123456789000.parquet
RESPONSE 403: 403 ...
ERROR CODE: AuthorizationPermissionMismatch

Cause: the Entra token was accepted, but the service principal has no blob write permission on the staging container. Typically it holds Owner or Contributor on the storage account. Those roles manage the account and do not include writing blobs. Like the credential itself, the role is first used on the first upload, not at startup.

Fix: in the Azure Portal open the storage account in account, or only the container in staging_container, then Access control (IAM) > Add role assignment, and assign Storage Blob Data Contributor to the application whose ID is in client_id. The role also covers deletes, which the target needs in order to remove each staged file after the load. Nothing is lost: the batch is redelivered until the role is in place, so expect the queue to grow meanwhile.

Other storage refusals arrive in the same shape, with a different code after ERROR CODE::

Error code	Cause	Fix
ContainerNotFound with 404	staging_container does not exist. The target never creates it	Create the container under Data storage > Containers, or correct the name
InvalidResourceName with 400	The container name has upper-case letters or other characters Azure does not allow	Use a lower-case container name
AuthorizationFailure with 403	The storage account firewall, virtual network rules or private endpoint typically reject the Director host's address	Allow that address under Networking. Allow the workspace as well, since the warehouse reads the staged file directly
AuthenticationFailed with 403	The host clock has drifted, or the token was issued for a different tenant	Check the clock on the Director host and confirm tenant_id
ServerBusy with 503, or AccountIsDisabled	The storage account is throttling ingress, or it is disabled	Raise timeout and lower max_size, or re-enable the account

All of these are retried until fixed, and nothing is dropped.

"record rejected by target" with a cast error from COPY INTO

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: failed to execute COPY command: record rejected by target: statement failed with state FAILED: BAD_REQUEST - [CAST_INVALID_INPUT] The value 'lots' of the type STRING cannot be cast to BIGINT
[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 deterministic failure for databricks-warehouse after 4 attempts — dropping (giving up): failed to execute COPY command: record rejected by target: statement failed with state FAILED: BAD_REQUEST - [CAST_INVALID_INPUT] The value 'lots' of the type STRING cannot be cast to BIGINT

Cause: the file reached the container, but the warehouse refused its contents. A value in the staged file does not fit the column in my_catalog.my_schema.my_table. Usually it is text where the table expects a number, a column the table does not have, or a schema that has drifted away from the table definition.

Fix: align the Avro or Parquet schema with the table, or normalize the field in a pipeline before it reaches the target. The message shown is the warehouse's own, and Query History holds the full statement next to it.

Data impact: this is the error that loses data. The rejection is deterministic, so the batch is dropped after four deliveries and the second line above is logged. The staged file is deleted as part of the failed load, so it cannot be inspected afterwards. Correct the schema before the fourth attempt if you need those events.

"statement failed with state: FAILED" with nothing after it

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 3 execute() failed for databricks-warehouse: failed to execute COPY command: statement failed with state: FAILED

Cause: the warehouse ended the statement in FAILED and answered with a code that Director does not treat as permanent, so the log line carries the state only. The detail lives in the workspace. Open the SQL warehouse's Query History, filter on the warehouse in http_path and on COPY INTO, and read the message on the failed statement. The causes that end here are, typically:

• The token's user is missing USE CATALOG, USE SCHEMA or MODIFY. The query history typically names PERMISSION_DENIED.
• my_catalog.my_schema.my_table does not exist. The target never creates tables. The query history typically names NOT_FOUND.
• No Unity Catalog external location and storage credential covers abfss://mycontainer@mystorageaccount.dfs.core.windows.net/, so the warehouse cannot read the file that Director just wrote.
• The warehouse is still starting. This one clears on its own.

Fix: grant the missing privilege, create the table, or create the external location, then let the next retry pick it up. Nothing is lost: the batch is redelivered until the load succeeds, and each attempt stages a fresh file and removes it again.

A capitalized "format" value stages JSON and the load fails

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: failed to execute COPY command: record rejected by target: statement failed with state FAILED: BAD_REQUEST - ...

Cause: format is accepted in any casing when the configuration is saved, but the writer that produces the file recognizes lower-case names only. A value such as PARQUET, Parquet or Avro passes validation, the file is written as JSON, and the COPY INTO statement still declares the format you asked for. The warehouse is handed JSON and told to read Parquet. For Parquet and Avro the schema requirement is skipped as well, so a missing schema is not reported either. csv, orc and text behave the same way in any casing. Nothing in the log names the format as the cause. The clue is that the staged file is JSON while the statement declares another format.

Fix: write format in lower case, and use only json, avro or parquet. Add schema for avro and parquet.

Data impact: every batch for that table fails the same way. Where the rejection is deterministic the batch is dropped after four deliveries, and the target's delivered counter never moves for that table.

"left in place for the running statement", or "polling timed out waiting for statement"

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 3 execute() failed for databricks-warehouse: failed to execute COPY command (staged blob mycontainer/databricks-staging/my_table/vmetric.1757590123456789000.parquet left in place for the running statement; it is not reaped automatically): warehouse statement outcome indeterminate: polling timed out waiting for statement 00000000-0000-0000-0000-000000000000: context deadline exceeded
[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 6). Reason: failed to test databricks connection: warehouse statement outcome indeterminate: polling timed out waiting for statement 00000000-0000-0000-0000-000000000000: context deadline exceeded

Cause: a statement did not reach a final state in time, so its outcome is unknown. The first line is a load that outran timeout, 300 seconds by default: a large file, a warehouse that is still scaling up, or a connection lost mid-statement. The second line is the startup check, which waits 10 seconds for SELECT 1. A stopped warehouse takes longer than that to come up, so every attempt fails until it is running. That message reads like a network fault, but the workspace did answer.

Fix: start the warehouse or enable auto-start on it. For the load, raise timeout and lower max_size or batch_size so each load is smaller. Look the statement up in Query History to see whether it finished after all.

Data impact: the startup case loses nothing, and data waits in the queue. The load case is the only error on this page that can duplicate rows. The batch is redelivered, the redelivery stages a new file and runs a second COPY INTO, and if the first statement commits in the meantime its rows are loaded twice. The file named in the message is kept on purpose, so that a statement still reading it does not lose its input, and Director does not return to delete it. Once the target is healthy again, reconcile the table for duplicates and delete the leftover files under staging_prefix.

"no such host", "context deadline exceeded", or certificate errors

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 4). Reason: failed to test databricks connection: warehouse statement outcome indeterminate: failed to execute request: Post "https://my-workspace.azuredatabricks.net/api/2.0/sql/statements": dial tcp: lookup my-workspace.azuredatabricks.net: no such host
[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: failed to upload to azure blob: Put "https://mystorageaccount.blob.core.windows.net/mycontainer/databricks-staging/my_table/vmetric.1757590123456789000.parquet": context deadline exceeded

Cause: Director could not complete an HTTPS request. The hostname in the message tells you which leg failed.

• no such host: on the workspace host, server_hostname is misspelled or was pasted with an https:// prefix or a trailing path. On the storage host, account is misspelled. Both take a bare name, not a URL.
• context deadline exceeded: the request did not finish within timeout seconds, 300 by default. The same value bounds the upload and the load, so a large file on a slow link and a slow warehouse both end here. Raise timeout, or lower max_size.
• tls: failed to verify certificate: x509: certificate signed by unknown authority: a TLS-intercepting proxy sits in the path. This target has no TLS options of its own and trusts the operating system certificate store, so install the proxy's CA certificate at the OS level.
• failed to parse response: invalid character '<' looking for beginning of value: something answered with HTML instead of JSON, typically a proxy login page.

Fix: allow outbound HTTPS (443) from the Director host to my-workspace.azuredatabricks.net, to mystorageaccount.blob.core.windows.net and to login.microsoftonline.com. If you use a proxy, set HTTPS_PROXY and NO_PROXY for the Director service. The workspace API and the Entra token request honor those variables, and the storage upload typically does as well. Nothing is lost. Retried until fixed.

"Failed to reinitialize target ... (attempt N)" with a configuration reason

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 2). Reason: failed to extract warehouse_id from http_path: /sql/1.0/endpoints/abc123def456

Reason	Fix
access_token is required for azdatabricks target, and the same message for server_hostname, http_path, catalog, account, staging_container, tenant_id, client_id and client_secret	Set the property. It is empty, or its ${ENV} variable is not exported to the Director service
failed to extract warehouse_id from http_path: ...	http_path must read /sql/1.0/warehouses/<id>. Copy the HTTP path from the warehouse's Connection details tab. The older /sql/1.0/endpoints/... form, a missing leading slash and a full URL are all rejected
invalid catalog name: ..., or invalid databricks schema (namespace) name: ...	catalog and namespace must be plain SQL identifiers: letters, digits and underscores, not starting with a digit. A hyphen, a dot or a backtick is rejected
invalid file format type: ...	format is not a name the target accepts. Use json, avro or parquet
at least one table must be configured (either 'table' or 'tables')	Set a catch-all table, or list tables
invalid table configuration: schema is required for parquet format, or the same for avro format	Set schema for that table, or at the top level for the catch-all
invalid table configuration: name is required	A tables entry has no name. Entries in tables have no default file name template, unlike the catch-all
schema file not found for: ..., or invalid schema format: ...	schema does not name a Library entry, a built-in schema, a readable file or valid inline JSON

Nothing is staged while the target fails to start, and nothing is lost. Fix the reason shown and the next attempt picks it up. Restarting Director is not required.

The target is healthy but nothing arrives in the table

Check these in order.

1. Events are dropped as unknown. With drop_unknown_stream_events at its default of true, an event whose table matches neither table nor a tables entry is discarded with no log line at all. Only the target's dropped counter moves. Add a catch-all table, or add the missing table to tables. Setting the flag to false does not deliver those events either. It turns the silent drop into file holder not found, which is redelivered every few seconds forever and stalls everything queued behind it on that worker.

2. debug.dont_send_logs is enabled. Events are processed and then dropped before anything is staged. Nothing is uploaded, nothing is loaded, and neither the delivered nor the dropped counter moves. With debug.status: true as well, startup logs once that log sending is disabled for the target. Remove the flag.

3. Every flush is failing. Search the Director log for execute() failed or Finalize failed on flush with the target name. The connection status stays positive for upload and load failures, because the startup check only talks to the warehouse. If the failures name one table only, check its format casing as described above.

Staged files build up in the container

Cause: Director deletes each staged file after the load, whether the load succeeded or failed for good. Two things leave files behind: a load whose outcome stayed unknown, which keeps its file on purpose, and a service principal that can write blobs but not delete them. A custom role with write but no delete uploads without any error and leaves every file.

Fix: check that the assignment is Storage Blob Data Contributor, which carries both rights. Add a lifecycle rule on the container to expire old files under staging_prefix, and delete the leftovers of the statements you have reconciled. Delivery is unaffected. The cost is storage only.