Databricks (S3 Staging)

Data Warehouse Target

Synopsis

The Databricks S3 target stages telemetry files to Amazon S3, then executes COPY INTO commands on Databricks SQL warehouses to load data into Unity Catalog tables.

Schema

- name: <string>
  description: <string>
  type: amazondatabricks
  properties:
    server_hostname: <string>
    http_path: <string>
    access_token: <string>
    catalog: <string>
    namespace: <string>
    staging_bucket: <string>
    staging_prefix: <string>
    region: <string>
    key: <string>
    secret: <string>
    session: <string>
    table: <string>
    schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    tables:
      - table: <string>
        schema: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
    batch_size: <integer>
    max_size: <integer>
    timeout: <integer>
    part_size: <integer>
    drop_unknown_stream_events: <boolean>
    field_format: <string>
    max_rows_per_rowgroup: <numeric>
    buffer_size: <numeric>
    data_page_version: <string>
    metadata: <key-value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be amazondatabricks
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Databricks Connection

Field	Required	Default	Description
server_hostname	Y	-	Databricks workspace URL (e.g., abc123.cloud.databricks.com)
http_path	Y	-	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123def456)
access_token	Y	-	Databricks personal access token
catalog	Y	-	Unity Catalog name
namespace	N	default	Databricks schema name

S3 Staging Configuration

Field	Required	Default	Description
staging_bucket	Y	-	S3 bucket name for staging files
staging_prefix	N	databricks-staging/	S3 prefix path
region	Y	-	AWS region for S3 bucket
key	N*	-	AWS access key ID (uses default credentials chain if omitted)
secret	N*	-	AWS secret access key
session	N	-	AWS session token for temporary credentials

* = key and secret take effect only when both are set. session applies only alongside them.

Table Configuration

Field	Required	Default	Description
table	Y*	-	Catch-all table name for all events
schema	Y*	-	Avro or Parquet schema: a Library name, a built-in name, or inline JSON. Parquet also accepts a schema file deployed under the schemas directory. Avro has no file lookup
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File naming template
format	N	json	File format. Use json, avro, or parquet — see the warning below.
compression	N	zstd	Compression algorithm
extension	N	-	File extension override. Defaults to the resolved format.
tables	N	-	Multiple table configurations (see below)
tables.table	Y	-	Target table name
tables.schema	Y*	-	Avro or Parquet schema for this table: a Library name, a built-in name, or inline JSON. Parquet also accepts a deployed schema file. Avro has no file lookup
tables.name	Y	-	File naming template for this table
tables.format	N	-	File format for this table. Falls back to the catch-all format.
tables.compression	N	-	Compression algorithm for this table. Falls back to the catch-all compression.
tables.extension	N	-	File extension override for this table. Falls back to the catch-all extension.

* = At least one of table (catch-all) or tables (multiple) must be configured. For Avro/Parquet formats, schema is required.

warning

Only json, avro, and parquet are written correctly. csv, orc, and text pass configuration validation, but the staged file is written as JSON while the COPY INTO command still declares the requested format, so the load fails.

Parquet Options

These apply only when format is parquet.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Maximum rows per Parquet row group
buffer_size	N	262144	Parquet page buffer size in bytes (256KB)
data_page_version	N	V2	Parquet data page version (V1, V2)
metadata	N	-	Key/value pairs written into the Parquet file metadata

Batch Configuration

Field	Required	Default	Description
batch_size	N	100000	Maximum events per file before flush
max_size	N	33554432	Maximum file size in bytes before flush (32MB)
timeout	N	300	COPY INTO command timeout in seconds
part_size	N	5	S3 multipart upload part size in MB. Values below 5 are raised to 5.
drop_unknown_stream_events	N	true	Drop events that do not match any configured table

Normalization

Field	Required	Default	Description
field_format	N	-	Apply format normalization (ECS, ASIM, UDM)

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Databricks

Details

Architecture Overview

The Databricks S3 target implements a two-stage loading pattern:

1. Stage Files to S3: Events are written to files in S3 using the configured format
2. Execute COPY INTO: SQL commands load data from S3 into Databricks Unity Catalog tables

Unity Catalog Integration

Catalog Structure:

• Tables are organized within Unity Catalog using three-level namespace: catalog.namespace.table
• The catalog field specifies the Unity Catalog name
• The namespace field specifies the schema (defaults to default)
• Table names are validated to ensure they are valid SQL identifiers

Warehouse ID Extraction:

• The target automatically extracts the warehouse ID from the http_path
• Example: /sql/1.0/warehouses/abc123def456 yields abc123def456
• This warehouse ID is used for all COPY INTO operations

Unity Catalog Permissions

The Databricks access token requires permissions to:

• Execute SQL statements on the specified warehouse
• Write data to the target catalog and schema
• Access the S3 staging location (configured separately in Databricks)

S3 Staging Operations

File Upload:

• Files are staged to s3://bucket/prefix/table/filename structure
• Uses AWS SDK multipart upload for large files
• Supports AWS credentials chain (access key, IAM role, instance profile)

Cleanup:

• Staged files are automatically deleted after successful COPY INTO execution
• A staged file is also deleted when the COPY INTO command fails outright
• A staged file is left in place only when the outcome of the COPY INTO command cannot be determined, for example when it outlives timeout. The log line names the object

File Format Support

Valid Formats:

• json: Newline-delimited JSON objects
• avro: Schema-based binary format, requires a schema
• parquet: Columnar storage format, requires a schema

csv, orc, and text pass configuration validation but are never written, as the warning above explains. Write every format value in lower case.

Schema Requirements:

• Avro and Parquet formats require schema field with valid schema definition
• Schema must match the expected table structure in Databricks
• JSON needs no schema

Multi-Table Routing

Catch-All Table:

• Use table field to send all events to a single table
• Simplest configuration for single-destination scenarios

Multiple Tables:

• Use tables array to route different event types to different tables
• Each table entry specifies table, schema, name, format fields
• Events routed based on SystemS3 field in pipeline

Example Configuration:

tables:
  - table: security_events
    schema: security_schema
    name: security.{{.Timestamp}}.parquet
    format: parquet
  - table: access_logs
    schema: access_schema
    name: access.{{.Timestamp}}.parquet
    format: parquet

Performance Considerations

Batch Processing:

• Events are buffered until batch_size or max_size limits are reached
• Larger batches reduce S3 API calls and COPY INTO operations
• Balance batch size against latency requirements

Upload Optimization:

• Multipart uploads automatically handle large files
• Configure part_size for optimal network performance
• Default part size is AWS SDK default (5 MB)

COPY INTO Performance:

• COPY INTO commands are executed with configurable timeout
• Failed COPY operations return errors for retry logic
• Warehouse must be running for COPY INTO to succeed

Warehouse State

Ensure the SQL warehouse is running before sending data. COPY INTO commands will fail if the warehouse is stopped. Configure warehouse auto-start or manual start procedures.

Error Handling

Upload Failures:

• Failed S3 uploads are retried based on sender configuration
• Permanent failures prevent COPY INTO execution
• Check S3 bucket permissions and network connectivity

COPY INTO Failures:

• Schema mismatches between files and tables cause failures
• Invalid SQL identifiers (catalog, schema, table names) are rejected at validation
• Check Databricks query history for detailed error messages

Examples

Basic Configuration

Sending telemetry to Databricks using S3 staging with Parquet format...

targets: - name: databricks-warehouse type: amazondatabricks properties: server_hostname: abc123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: production_data namespace: telemetry staging_bucket: datastream-staging region: us-east-1 table: events schema: event_schema name: events.{{.Timestamp}}.parquet format: parquet

With AWS Credentials

Using explicit AWS credentials for S3 staging access...

targets: - name: databricks-secure type: amazondatabricks properties: server_hostname: xyz789.cloud.databricks.com http_path: /sql/1.0/warehouses/xyz789def123 access_token: "${DATABRICKS_TOKEN}" catalog: security_analytics namespace: logs staging_bucket: security-logs-staging staging_prefix: databricks/ region: us-west-2 key: "${AWS_ACCESS_KEY}" secret: "${AWS_SECRET_KEY}" table: security_events schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet

Multi-Table Configuration

Routing different event types to separate Databricks tables...

targets: - name: databricks-multi-table type: amazondatabricks properties: server_hostname: abc123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: analytics namespace: production staging_bucket: analytics-staging region: us-east-1 tables: - table: authentication_events schema: auth_schema name: auth.{{.Timestamp}}.parquet format: parquet - table: network_events schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet - table: application_logs schema: app_schema name: app.{{.Timestamp}}.parquet format: parquet

High-Volume Configuration

Optimizing for high-volume ingestion with batch limits and compression...

targets: - name: databricks-high-volume type: amazondatabricks properties: server_hostname: abc123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: high_volume_data namespace: streaming staging_bucket: streaming-staging region: us-east-1 batch_size: 100000 max_size: 134217728 part_size: 16 timeout: 600 table: streaming_events schema: streaming_schema name: stream.{{.Timestamp}}.parquet format: parquet compression: snappy

JSON Format

Using JSON format for flexible schema evolution and debugging...

targets: - name: databricks-json type: amazondatabricks properties: server_hostname: abc123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: development namespace: test_data staging_bucket: dev-staging region: us-east-1 table: test_events name: test.{{.Timestamp}}.json format: json

With Normalization

Applying ECS normalization before loading to Databricks...

targets: - name: databricks-normalized type: amazondatabricks properties: server_hostname: abc123.cloud.databricks.com http_path: /sql/1.0/warehouses/abc123def456 access_token: "${DATABRICKS_TOKEN}" catalog: security_data namespace: normalized staging_bucket: security-staging region: us-east-1 field_format: ECS table: ecs_events schema: ecs_schema name: ecs.{{.Timestamp}}.parquet format: parquet

Production Configuration

Production-ready configuration with performance tuning, AWS credentials, and multi-table routing...

targets: - name: databricks-production type: amazondatabricks properties: server_hostname: production.cloud.databricks.com http_path: /sql/1.0/warehouses/prod123abc456 access_token: "${DATABRICKS_TOKEN}" catalog: production_analytics namespace: telemetry staging_bucket: production-staging-bucket staging_prefix: datastream/databricks/ region: us-east-1 key: "${AWS_ACCESS_KEY}" secret: "${AWS_SECRET_KEY}" batch_size: 50000 max_size: 67108864 part_size: 10 timeout: 300 field_format: ASIM tables: - table: security_events schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet compression: snappy - table: audit_logs schema: audit_schema name: audit.{{.Timestamp}}.parquet format: parquet compression: snappy - table: network_flows schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet compression: snappy

Troubleshooting

This section covers the errors you are most likely to see with the amazondatabricks target, what causes each one, and how to fix it. Delivery happens in two steps, and each step fails in its own way. Director stages a file to Amazon S3 with your AWS credentials, then runs COPY INTO on the SQL warehouse with your Databricks access token. The step named in the error tells you which of the two to look at.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.
• The workspace's Query History, for anything that happens after a statement reaches the warehouse. The log line carries the statement ID and its final state, and Databricks keeps the detail.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission is missing?

Three separate grants are involved, and none of them covers for another.

• The AWS credentials in key and secret, or the host's default credentials when either field is empty, upload the staged file into staging_bucket.
• The Databricks access token in access_token runs COPY INTO on the warehouse named by http_path.
• The warehouse reads the staged file back with its own access to the bucket, which is configured on the Databricks side. Your key and secret are never handed to it.

At startup Director proves the AWS credential with an identity call that touches no bucket, so a missing s3:PutObject is not discovered until the first flush. A target that reports connected has not yet proven that it can write to the bucket.

Error text	The right that is missing	Where it is granted
failed to upload to s3: ... api error AccessDenied: ...	s3:PutObject	The AWS identity, on arn:aws:s3:::my-bucket/<staging prefix>*
A multipart upload fails and leaves parts in the bucket	s3:AbortMultipartUpload	The AWS identity, same prefix
Staged files are never removed after a successful load	s3:DeleteObject	The AWS identity, same prefix
failed to upload to s3: ... api error AccessDenied: ... on a bucket whose default encryption is SSE-KMS	kms:GenerateDataKey and kms:Decrypt, typically	The key that encrypts the bucket
databricks API error (status 403): {"error_code":"PERMISSION_DENIED", ...}	CAN USE on the SQL warehouse	Databricks, to the identity that owns the access token
reached terminal state: FAILED with no reason, and Query History reports a privilege error	USE CATALOG on my_catalog, USE SCHEMA on my_schema, MODIFY and SELECT on my_catalog.my_schema.my_table, typically	Unity Catalog, to the identity that owns the access token
reached terminal state: FAILED with no reason, and Query History reports that the file could not be read	Read access to s3://my-bucket from the warehouse	Databricks, as a storage credential and external location, or an instance profile, with s3:GetObject and s3:ListBucket

note

The target never creates the table. my_catalog.my_schema.my_table must already exist, with columns that match the file you stage.

"warehouse statement outcome indeterminate" when the target starts

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 9). Reason: failed to test databricks connection: warehouse statement outcome indeterminate: polling timed out waiting for statement <statement id>: context deadline exceeded

Cause: expect this one first. A stopped warehouse is the usual reason. At startup Director runs a SELECT 1 against the warehouse and allows it 10 seconds. A stopped warehouse accepts the statement and then starts itself, which takes minutes, so the budget expires before any answer arrives and the outcome cannot be determined.

Fix: check the warehouse before anything else.

1. Open the SQL warehouse named by http_path in the workspace and read its state. A warehouse that is Starting needs no fix. Wait, and one of the next attempts succeeds on its own.
2. If it is Stopped, start it. For a target that receives data continuously, use a serverless warehouse or raise the Auto stop setting so the warehouse is not stopped between flushes.
3. If the warehouse is running and the message persists, the workspace is answering slowly or a proxy is holding the connection. See the network entry below.

Retries back off from 5 seconds to once a minute and continue until the warehouse answers, so the attempt counter grows for as long as the cause lasts. A high count is not a separate fault. Nothing is lost. Incoming data waits in the Director queue and is delivered once the target starts.

"databricks API error (status 403)" or "(status 401)"

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 1). Reason: failed to test databricks connection: databricks API error (status 403): {"error_code":"PERMISSION_DENIED","message":"invalid access token"}

Cause: the workspace refused the value in access_token. A 403 with PERMISSION_DENIED typically means the token is wrong, or that its owner has no CAN USE on the warehouse. A 401 with UNAUTHENTICATED typically means the token has expired or was revoked. The same message appears at flush time, prefixed with failed to execute COPY command:, when a token that was working expires while the target runs. A workspace IP access list refuses in the same shape.

Fix: generate a personal access token in the workspace under Settings > Developer > Access tokens, and grant its owner CAN USE on the warehouse. Confirm that a ${DATABRICKS_TOKEN} reference, or a $secret{...} reference, resolves for the Director service and not only in the shell you tested from, because an unresolved reference arrives as an empty token. If the workspace has an IP access list, add the Director host to it.

Nothing is lost. Data stays queued and is delivered once the token is accepted.

"failed to upload to s3" with "AccessDenied"

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 2 execute() failed for databricks-warehouse: target broken: failed to finalize target cache: failed to upload to s3: operation error S3: PutObject, https response error StatusCode: 403, RequestID: ..., api error AccessDenied: ...

Cause: the upload into the staging bucket was refused. The target started normally, because the startup check only proves that an AWS credential exists. This is the first moment the bucket itself is touched. The identity has no s3:PutObject on the prefix, or a bucket policy denies it, or the bucket enforces SSE-KMS and the identity cannot use the key.

Fix: grant s3:PutObject, s3:DeleteObject, and s3:AbortMultipartUpload on arn:aws:s3:::my-bucket/<staging prefix>* to the identity behind key and secret. Both fields take effect only when both are set, so when either is empty the host's default credentials are used instead and the grant belongs to that identity. For a KMS-encrypted bucket, allow kms:GenerateDataKey and kms:Decrypt on the key. Two neighboring failures name the bucket rather than a permission: api error NoSuchBucket: ... means staging_bucket is misspelled, and a 301 with PermanentRedirect, typically, means region is not the region the bucket lives in.

Nothing is lost. The batch is discarded from memory, the payloads stay queued, and they are redelivered until the upload succeeds.

"reached terminal state: FAILED" with a value that cannot be cast

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: failed to execute COPY command: record rejected by target: statement <statement id> reached terminal state: FAILED: BAD_REQUEST - [CAST_INVALID_INPUT] The value 'lots' of the type STRING cannot be cast to BIGINT

Cause: the file reached the warehouse and COPY INTO refused its rows. The message names the offending value and both types. The staged file and the table disagree: a field carries text where the column is numeric, a column is missing, or a Parquet schema does not line up with the table. Databricks answers this as a permanent rejection, so it is not retried indefinitely.

Fix: align the two. Compare the schema configured for the table against the column types of my_catalog.my_schema.my_table, then widen the column or correct the field type. Where a pipeline can emit either text or a number in the same field, make that column STRING in Databricks. Confirm the table exists, because COPY INTO does not create it.

Records are lost. The batch is redelivered four times and then given up, logged as deterministic failure for databricks-warehouse after 4 attempts — dropping (giving up), and the staged object is deleted. Fix the mismatch before the fourth delivery, or re-send the data from the source afterwards.

"format" written in capitals stages the wrong file

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: failed to execute COPY command: record rejected by target: statement <statement id> reached terminal state: FAILED: BAD_REQUEST - ...

Cause: format is validated without regard to case, but only a lower-case value produces the format it names. Parquet and PARQUET therefore pass validation, skip the schema requirement that parquet enforces, and the file is staged as JSON lines while COPY INTO still declares FILEFORMAT = PARQUET. The warehouse reads a JSON file as Parquet and refuses it. csv, orc, and text pass validation the same way and end the same way, because none of them is ever written.

Fix: write format in lower case, and use only json, avro, or parquet. Set it in every entry under tables as well, because an entry with no format of its own inherits the catch-all value.

Records are lost. The rejection is the same one as above, so the batch is dropped after four deliveries.

warning

This trap is easy to miss, because nothing looks wrong until the load runs. The configuration is accepted, the target reports connected, and the file is staged.

"reached terminal state: FAILED" with no reason after it

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 execute() failed for databricks-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: statement <statement id> reached terminal state: FAILED

Cause: the statement reached the warehouse and failed there. The log line carries the statement ID and the final state, and the workspace holds the reason. Open Query History, find the statement by that ID, and read the error recorded against it. What turns up is usually one of these.

• The access token lacks USE CATALOG, USE SCHEMA, or MODIFY on the table.
• The warehouse cannot read s3://my-bucket, because no storage credential, external location, or instance profile covers the staging bucket.
• my_catalog, my_schema, or my_table does not exist, or is spelled differently in the configuration.
• The warehouse is still starting, or its query queue is full.

Fix: grant or correct whatever Query History names, then wait for the next attempt. No restart is needed. Nothing is lost. The batch is discarded from memory, the payloads stay queued, and they are redelivered on every attempt until the statement succeeds.

tip

Filter Query History by the warehouse in http_path and look for statements that begin with COPY INTO. Their text names the table and the staged object, which tells you which table configuration produced the failure.

"warehouse statement outcome indeterminate" during COPY INTO

[Error] [director] [target-<target id>] [databricks-warehouse] Sender worker 1 Finalize failed on flush for target "databricks-warehouse": failed to execute COPY command (staged object s3://my-bucket/databricks-staging/my_table/vmetric.1757590000123456789.parquet left in place for the running statement; it is not reaped automatically): warehouse statement outcome indeterminate: polling timed out waiting for statement <statement id>: context deadline exceeded

Cause: COPY INTO ran longer than timeout, which is 300 seconds by default, so Director stopped waiting before the warehouse answered. The same wrapper appears when the connection drops in the middle of a statement. The load may well have finished afterwards, and that cannot be told from here, which is why the object named in the message is left in place rather than deleted.

Fix: give the statement room. Raise timeout, lower max_size so each file is smaller, and keep the warehouse warm so a cold start is not charged against the budget. A busy warehouse queues statements, so a larger warehouse, or fewer targets pointed at it, helps as well.

Rows can be duplicated. The batch is redelivered, and the redelivery stages a new object and runs a second COPY INTO. If the first one had in fact committed, its rows are in the table twice. De-duplicate in Databricks after an episode of this, and set a lifecycle rule on staging_prefix so the objects left behind expire on their own.

"no such host", "connection refused", or "certificate signed by unknown authority"

[Error] [director] [target-<target id>] [databricks-warehouse] Failed to reinitialize target "databricks-warehouse" (attempt 2). Reason: failed to test databricks connection: warehouse statement outcome indeterminate: failed to execute request: Post "https://https//my-workspace.cloud.databricks.com/api/2.0/sql/statements": dial tcp: lookup https: no such host

Cause: Director could not reach the workspace or Amazon.

• lookup https: no such host is the example above. server_hostname was pasted with a scheme in front of it. The value must be the bare host, my-workspace.cloud.databricks.com, with no https:// and no path.
• no such host on the real hostname, connection refused, or context deadline exceeded points at DNS or a firewall.
• tls: failed to verify certificate: x509: certificate signed by unknown authority means an intercepting proxy or a private CA is in the path. This target reads no TLS options, so there is no setting to trust a CA or to skip verification.
• failed to parse response: invalid character '<' looking for beginning of value means something other than the workspace answered, typically a proxy login page or an SSO redirect.

Fix: correct server_hostname to the bare host, then allow outbound HTTPS on 443 from the Director host to my-workspace.cloud.databricks.com, to sts.<region>.amazonaws.com, and to the bucket endpoint my-bucket.s3.<region>.amazonaws.com. Install a private CA certificate in the host's own trust store, or exclude the workspace host from interception. Where an outbound proxy is required, set HTTPS_PROXY for the Director service and list internal hosts in NO_PROXY. Nothing is lost, and the target retries until the workspace answers.

"Failed to reinitialize target ... (attempt N)" for a configuration mistake

Some settings are only checked when the target starts, so a mistake in them arrives through the reinitialize loop rather than as a validation error. The target retries a configuration that cannot work until you change it, and nothing is sent while that lasts.

Reason text	Fix
failed to extract warehouse_id from http_path: ...	http_path must be exactly /sql/1.0/warehouses/<warehouse id>, leading slash included. Copy it from the warehouse's Connection details tab. A pasted JDBC URL, or the older /sql/1.0/endpoints/... form, is not accepted
invalid catalog name: ..., invalid databricks schema (namespace) name: ...	catalog and namespace must be plain SQL identifiers: letters, digits, and underscores, not starting with a digit. A hyphen, dot, space, or backtick is refused
invalid table name: ..., invalid catch-all table name: ...	The same rule applies to table and to every tables.table
invalid table configuration: name is required	Every entry under tables needs its own name. Only the catch-all name has a default
invalid table configuration: schema is required for parquet format	Add schema wherever format is avro or parquet, including entries that inherit the format from the catch-all
invalid schema format: invalid field format: ...	The schema value matched no library entry and no deployed file, so it was read as an inline field list and failed on the first token. For parquet, deploy the file under the schemas directory. For avro, a file name is never looked up: use a library name, a built-in name, or inline JSON
failed to generate unique file path after 10 attempts for thread 0	Two tables resolve to the same file name. Keep {{.Timestamp}} in every name
databricks API error (status 404)	The warehouse in http_path does not exist in the workspace in server_hostname. Both must come from the same workspace
databricks API error (status 429)	The workspace is rate limiting. Flush less often by raising batch_size and max_size

The target is healthy but no rows arrive

Nothing fails in these cases, so there is no error to search for. Check them in order.

1. Events are dropped as unknown tables. With drop_unknown_stream_events at its default of true, an event whose table matches no entry under tables, with no catch-all table to fall back on, is discarded without a log line. Only the target's dropped counter moves. Add the table, add a catch-all table, or set drop_unknown_stream_events: false to turn the silent drop into a visible file holder not found error.

2. debug.dont_send_logs is enabled. Events are processed by the pipeline and then discarded before anything is buffered. Nothing is staged, no counter moves, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well. Remove the flag when you have finished testing.

   Log sending is disabled for this target (databricks-warehouse). Logs will be processed by the pipeline but will not be sent to the target.

3. The rows arrive later than expected. A table is loaded only when its file is closed, which happens at batch_size records or max_size bytes. On a low-volume stream that can be a long wait. Lower both to shorten it.

4. Staged files pile up under the prefix. The staged object is deleted after a successful load, and a refused delete is not reported. Without s3:DeleteObject the prefix fills up quietly while the data itself arrives normally. Grant the action, and set a lifecycle rule on staging_prefix as a safety net.

5. The staged files are not compressed. For json, only gzip is applied, so the default of zstd stages uncompressed files. avro and parquet use their own codecs, where snappy and zstd are both recognized.