Databricks (Azure Blob Storage)

Data Warehouse Target

Synopsis

The Databricks Azure Blob target stages telemetry files to Azure Blob Storage, then executes COPY INTO commands on Databricks SQL warehouses to load data into Unity Catalog tables.

Schema

- name: <string>
  description: <string>
  type: azdatabricks
  properties:
    server_hostname: <string>
    http_path: <string>
    access_token: <string>
    catalog: <string>
    namespace: <string>
    account: <string>
    staging_container: <string>
    staging_prefix: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    table: <string>
    schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    tables:
      - table: <string>
        schema: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
    batch_size: <integer>
    max_size: <integer>
    timeout: <integer>
    drop_unknown_stream_events: <boolean>
    field_format: <string>
    max_rows_per_rowgroup: <numeric>
    buffer_size: <numeric>
    data_page_version: <string>
    metadata: <key-value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be azdatabricks
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Databricks Connection

Field	Required	Default	Description
server_hostname	Y	-	Databricks workspace URL (e.g., abc123.azuredatabricks.net)
http_path	Y	-	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123def456)
access_token	Y	-	Databricks personal access token
catalog	Y	-	Unity Catalog name
namespace	N	default	Databricks schema name

Azure Blob Staging Configuration

Field	Required	Default	Description
account	Y	-	Azure storage account name
staging_container	Y	-	Azure Blob container name for staging files
staging_prefix	N	databricks-staging/	Blob prefix path
tenant_id	Y	-	Azure AD tenant ID
client_id	Y	-	Service principal client ID
client_secret	Y	-	Service principal client secret

Table Configuration

Field	Required	Default	Description
table	Y*	-	Catch-all table name for all events
schema	Y*	-	Avro/Parquet schema: a Library name, built-in name, file path, or inline JSON definition
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File naming template
format	N	json	File format. Use json, avro, or parquet — see the warning below.
compression	N	zstd	Compression algorithm
extension	N	-	File extension override. Defaults to the resolved format.
tables	N	-	Multiple table configurations (see below)
tables.table	Y	-	Target table name
tables.schema	Y*	-	Avro/Parquet schema for this table (Library name, built-in name, file path, or inline JSON)
tables.name	Y	-	File naming template for this table
tables.format	N	-	File format for this table. Falls back to the catch-all format.
tables.compression	N	-	Compression algorithm for this table. Falls back to the catch-all compression.
tables.extension	N	-	File extension override for this table. Falls back to the catch-all extension.

* = At least one of table (catch-all) or tables (multiple) must be configured. For Avro/Parquet formats, schema is required.

warning

Only json, avro, and parquet are written correctly. csv, orc, and text pass configuration validation, but the staged file is written as JSON while the COPY INTO command still declares the requested format, so the load fails.

Parquet Options

These apply only when format is parquet.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Maximum rows per Parquet row group
buffer_size	N	262144	Parquet page buffer size in bytes (256KB)
data_page_version	N	V2	Parquet data page version (V1, V2)
metadata	N	-	Key/value pairs written into the Parquet file metadata

Batch Configuration

Field	Required	Default	Description
batch_size	N	100000	Maximum events per file before flush
max_size	N	33554432	Maximum file size in bytes before flush (32MB)
timeout	N	300	COPY INTO command timeout in seconds
drop_unknown_stream_events	N	true	Drop events that do not match any configured table

Normalization

Field	Required	Default	Description
field_format	N	-	Apply format normalization (ECS, ASIM, UDM)

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Databricks

Details

Architecture Overview

The Databricks Azure Blob target implements a two-stage loading pattern:

1. Stage Files to Azure Blob: Events are written to files in Azure Blob Storage using the configured format
2. Execute COPY INTO: SQL commands load data from Blob Storage into Databricks Unity Catalog tables using ABFSS paths

Unity Catalog Integration

Catalog Structure:

• Tables are organized within Unity Catalog using three-level namespace: catalog.namespace.table
• The catalog field specifies the Unity Catalog name
• The namespace field specifies the schema (defaults to default)
• Table names are validated to ensure they are valid SQL identifiers

Warehouse ID Extraction:

• The target automatically extracts the warehouse ID from the http_path
• Example: /sql/1.0/warehouses/abc123def456 → abc123def456
• This warehouse ID is used for all COPY INTO operations

Unity Catalog Permissions

The Databricks access token requires permissions to:

• Execute SQL statements on the specified warehouse
• Write data to the target catalog and schema
• Access the Azure Blob staging location (configured separately in Databricks)

Azure Blob Staging Operations

File Upload:

• Files are staged to https://{account}.blob.core.windows.net/{container}/{prefix}/{table}/{filename} structure
• Uses Azure SDK for secure uploads with service principal authentication
• Supports Azure AD authentication through client credentials

ABFSS Path Construction:

• The target automatically constructs ABFSS paths for COPY INTO commands
• Format: abfss://{container}@{account}.dfs.core.windows.net/{prefix}/{table}/{filename}
• ABFSS protocol is used for direct Databricks access to Azure Data Lake Storage Gen2

Cleanup:

• Staged files are automatically deleted after successful COPY INTO execution
• Failed uploads remain in Blob Storage for troubleshooting

Service Principal Authentication

Azure AD Integration:

• Uses service principal (client credentials) for Azure Blob Storage authentication
• Requires tenant_id, client_id, and client_secret configuration
• Service principal must have Storage Blob Data Contributor role on the container

Required Permissions:

• Storage Blob Data Contributor: Write and delete blobs in staging container
• Storage Blob Data Reader: Optional, for Databricks direct access

Service Principal Permissions

Ensure the service principal has appropriate permissions on both the staging container (for DataStream uploads) and the Databricks workspace (for COPY INTO access).

File Format Support

Valid Formats:

• CSV: Comma-separated values with optional headers
• JSON: Newline-delimited JSON objects
• AVRO: Schema-based binary format (requires schema)
• ORC: Optimized row columnar format
• PARQUET: Columnar storage format (requires schema)
• TEXT: Plain text with delimiters

Schema Requirements:

• Avro and Parquet formats require schema field with valid schema definition
• Schema must match the expected table structure in Databricks
• Other formats use schema inference from data

Multi-Table Routing

Catch-All Table:

• Use table field to send all events to a single table
• Simplest configuration for single-destination scenarios

Multiple Tables:

• Use tables array to route different event types to different tables
• Each table entry specifies table, schema, name, format fields
• Events routed based on SystemS3 field in pipeline

Example Configuration:

tables:
  - table: security_events
    schema: security_schema.avsc
    name: security.{{.Timestamp}}.parquet
    format: parquet
  - table: access_logs
    schema: access_schema.avsc
    name: access.{{.Timestamp}}.parquet
    format: parquet

Performance Considerations

Batch Processing:

• Events are buffered until batch_size or max_size limits are reached
• Larger batches reduce Blob API calls and COPY INTO operations
• Balance batch size against latency requirements

Upload Optimization:

• Azure SDK automatically handles large blob uploads
• Uses block blobs for efficient data transfer
• Connection pooling optimizes network performance

COPY INTO Performance:

• COPY INTO commands are executed with configurable timeout
• Failed COPY operations return errors for retry logic
• Warehouse must be running for COPY INTO to succeed

Warehouse State

Ensure the SQL warehouse is running before sending data. COPY INTO commands will fail if the warehouse is stopped. Configure warehouse auto-start or manual start procedures.

Error Handling

Upload Failures:

• Failed Blob uploads are retried based on sender configuration
• Permanent failures prevent COPY INTO execution
• Check service principal permissions and network connectivity

COPY INTO Failures:

• Schema mismatches between files and tables cause failures
• Invalid SQL identifiers (catalog, schema, table names) are rejected at validation
• Check Databricks query history for detailed error messages

Examples

Basic Configuration

Sending telemetry to Databricks using Azure Blob staging with Parquet format...

targets: - name: databricks-warehouse type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: production_data namespace: telemetry account: datastreamstaging staging_container: databricks-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: events schema: event_schema.avsc name: events.{{.Timestamp}}.parquet format: parquet

With Custom Staging Prefix

Using custom blob prefix for organized staging file structure...

targets: - name: databricks-organized type: azdatabricks properties: server_hostname: xyz789.azuredatabricks.net http_path: /sql/1.0/warehouses/xyz789def123 access_token: "${DATABRICKS_TOKEN}" catalog: security_analytics namespace: logs account: securitystorage staging_container: staging staging_prefix: datastream/databricks/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: security_events schema: security_schema.avsc name: security.{{.Timestamp}}.parquet format: parquet

Multi-Table Configuration

Routing different event types to separate Databricks tables...

targets: - name: databricks-multi-table type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: analytics namespace: production account: analyticsstorage staging_container: staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" tables: - table: authentication_events schema: auth_schema.avsc name: auth.{{.Timestamp}}.parquet format: parquet - table: network_events schema: network_schema.avsc name: network.{{.Timestamp}}.parquet format: parquet - table: application_logs schema: app_schema.avsc name: app.{{.Timestamp}}.parquet format: parquet

High-Volume Configuration

Optimizing for high-volume ingestion with batch limits and compression...

targets: - name: databricks-high-volume type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: high_volume_data namespace: streaming account: streamingstorage staging_container: high-volume-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 100000 max_size: 134217728 timeout: 600 table: streaming_events schema: streaming_schema.avsc name: stream.{{.Timestamp}}.parquet format: parquet compression: snappy

JSON Format

Using JSON format for flexible schema evolution and debugging...

targets: - name: databricks-json type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: development namespace: test_data account: devstorage staging_container: dev-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: test_events name: test.{{.Timestamp}}.json format: json

With Normalization

Applying ASIM normalization before loading to Databricks...

targets: - name: databricks-normalized type: azdatabricks properties: server_hostname: abc123.azuredatabricks.net http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: security_data namespace: normalized account: securitystorage staging_container: security-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" field_format: ASIM table: asim_events schema: asim_schema.avsc name: asim.{{.Timestamp}}.parquet format: parquet

Production Configuration

Production-ready configuration with performance tuning and multi-table routing...

targets: - name: databricks-production type: azdatabricks properties: server_hostname: production.azuredatabricks.net http_path: /sql/1.0/warehouses/prod123abc456 access_token: "${DATABRICKS_TOKEN}" catalog: production_analytics namespace: telemetry account: productionstorage staging_container: production-staging staging_prefix: datastream/databricks/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 50000 max_size: 67108864 timeout: 300 field_format: ECS tables: - table: security_events schema: security_schema.avsc name: security.{{.Timestamp}}.parquet format: parquet compression: snappy - table: audit_logs schema: audit_schema.avsc name: audit.{{.Timestamp}}.parquet format: parquet compression: snappy - table: network_flows schema: network_schema.avsc name: network.{{.Timestamp}}.parquet format: parquet compression: snappy