Snowflake (S3 Staging)

Data Warehouse Target

Synopsis

The Snowflake S3 target stages telemetry files to Amazon S3, then executes COPY INTO commands on Snowflake to load data into tables.

Schema

- name: <string>
  description: <string>
  type: amazonsnowflake
  properties:
    account: <string>
    username: <string>
    password: <string>
    database: <string>
    schema: <string>
    warehouse: <string>
    role: <string>
    staging_bucket: <string>
    staging_prefix: <string>
    region: <string>
    key: <string>
    secret: <string>
    session: <string>
    table: <string>
    data_schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    tables:
      - table: <string>
        schema: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
    batch_size: <integer>
    max_size: <integer>
    timeout: <integer>
    part_size: <integer>
    drop_unknown_stream_events: <boolean>
    field_format: <string>
    max_rows_per_rowgroup: <numeric>
    buffer_size: <numeric>
    data_page_version: <string>
    metadata: <key-value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be amazonsnowflake
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Snowflake Connection

Field	Required	Default	Description
account	Y	-	Snowflake account identifier (e.g., abc123.us-east-1)
username	Y	-	Snowflake username
password	Y	-	Snowflake password
database	Y	-	Snowflake database name
schema	N	PUBLIC	Snowflake schema name. Must be a valid SQL identifier — letters, digits, and underscores only, not starting with a digit.
warehouse	N	-	Snowflake virtual warehouse name
role	N	-	Snowflake role name

S3 Staging Configuration

Field	Required	Default	Description
staging_bucket	Y	-	S3 bucket name for staging files
staging_prefix	N	snowflake-staging/	S3 prefix path
region	Y	-	AWS region for S3 bucket
key	N*	-	AWS access key ID (uses default credentials chain if omitted)
secret	N*	-	AWS secret access key
session	N	-	AWS session token for temporary credentials

* = key and secret take effect only when both are set. session applies only alongside them.

Table Configuration

Field	Required	Default	Description
table	Y*	-	Catch-all table name for all events
data_schema	Y*	-	Avro/Parquet schema for the catch-all table: a Library name, a built-in name, an inline JSON definition, or a field:type,... list. When it is empty, the top-level schema is read in its place, which still has to be a valid SQL identifier because it is also the Snowflake schema.
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File naming template
format	N	json	File format. Use json, avro, or parquet — see the warning below.
compression	N	zstd	Compression algorithm
extension	N	-	File extension override. Defaults to the resolved format.
tables	N	-	Multiple table configurations (see below)
tables.table	Y	-	Target table name
tables.schema	Y*	-	Avro/Parquet schema for this table: a Library name, a built-in name, an inline JSON definition, or a field:type,... list. A file name such as event_schema.avsc is not looked up, so it fails as an invalid field list. This table does not read data_schema.
tables.name	Y	-	File naming template for this table
tables.format	N	-	File format for this table. Falls back to the catch-all format.
tables.compression	N	-	Compression algorithm for this table. Falls back to the catch-all compression.
tables.extension	N	-	File extension override for this table. Falls back to the catch-all extension.

warning

Only json, avro, and parquet are written correctly. csv, orc, and xml pass configuration validation, but the staged file is written as JSON while the COPY INTO command still declares the requested format, so the load fails.

Parquet Options

These apply only when format is parquet.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Maximum rows per Parquet row group
buffer_size	N	262144	Page buffer size in bytes (256KB) of the older Parquet writer only
data_page_version	N	-	V1, or no value, uses the column-by-column writer, which writes version 1 data pages, unless the schema or the compression keeps the older writer. V2 keeps the older Parquet writer, which writes version 2 data pages. Before, the default was V2
metadata	N	-	Key/value pairs written into the Parquet file metadata

How the staged files are written, which values they keep, and which outputs still use the older writer: see How Targets Write Parquet.

* = At least one of table (catch-all) or tables (multiple) must be configured. For Avro/Parquet formats, the catch-all table needs data_schema (or, failing that, schema) and every tables entry needs its own schema.

Batch Configuration

Field	Required	Default	Description
batch_size	N	100000	Maximum events per file before flush
max_size	N	33554432	Maximum file size in bytes before flush (32MB)
timeout	N	300	Seconds allowed for the staging upload and for the COPY INTO command
part_size	N	5	S3 multipart upload part size in MB. Values below 5 are raised to 5.
drop_unknown_stream_events	N	true	Drop events that do not match any configured table

Normalization

Field	Required	Default	Description
field_format	N	-	Data normalization format. Any supported format is accepted; OCSF is a common choice for security data in a lakehouse. See applicable Normalization section

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Snowflake

Details

Architecture Overview

The Snowflake S3 target implements a two-stage loading pattern:

1. Stage Files to S3: Events are written to files in S3 using the configured format
2. Execute COPY INTO: SQL commands load data from S3 into Snowflake tables

Snowflake Connection

Account identifier:

• Format: <account_locator>.<region> (e.g., abc123.us-east-1)
• Account locator is visible in your Snowflake URL
• Region is the cloud region where your Snowflake account is deployed

Authentication:

• Uses username/password authentication
• Credentials are used to connect to Snowflake SQL API v2
• Supports optional warehouse and role specification

Database and Schema:

• Database name is required and must be a valid SQL identifier
• Schema defaults to PUBLIC if not specified
• Both database and schema names are validated for SQL compliance

Snowflake Permissions

The Snowflake user requires permissions to:

• Execute SQL statements using the specified warehouse
• Write data to the target database and schema

Staging Bucket Access

The COPY INTO command names the staged file as a plain s3:// URL. It sends no storage integration, no credentials and no named stage, so Snowflake has to be able to read the staging bucket on its own. The AWS key and secret upload the file and are never passed on to Snowflake. Before sending data, give the Snowflake account read access to the staging prefix on the bucket itself. No target setting passes a storage integration or credentials to the command, so a bucket that can only be read with them cannot be loaded from. When Snowflake cannot read the object, every load fails with 091003 Failure using stage area. Cause: [Access Denied] and no rows arrive.

S3 Staging Operations

File Upload:

• Files are staged to s3://bucket/prefix/table/filename structure
• Uses AWS SDK multipart upload for large files
• Supports AWS credentials chain (access key, IAM role, instance profile)

Cleanup:

• Staged files are automatically deleted after successful COPY INTO execution
• A staged file whose COPY INTO is refused is deleted as well, so a failed load leaves nothing behind to inspect
• The exception is a load whose outcome stays unknown, such as a COPY INTO that runs past timeout. That object is kept, and the retry stages a second copy of the same batch

File Format Support

Valid Formats:

• CSV: Comma-separated values with optional headers
• JSON: Newline-delimited JSON objects
• AVRO: Schema-based binary format (requires schema)
• ORC: Optimized row columnar format
• PARQUET: Columnar storage format (requires schema)
• XML: XML document format

Schema Requirements:

• Avro and Parquet formats require a schema definition: data_schema for the catch-all table, schema on each tables entry
• Schema must match the expected table structure in Snowflake
• Other formats use schema inference from data

Multi-Table Routing

Catch-All Table:

• Use table field to send all events to a single table
• Simplest configuration for single-destination scenarios

Multiple Tables:

• Use tables array to route different event types to different tables
• Each table entry specifies table, schema, name, format fields
• Events routed based on SystemS3 field in pipeline

Example Configuration:

tables:
  - table: security_events
    schema: security_schema
    name: security.{{.Timestamp}}.parquet
    format: parquet
  - table: access_logs
    schema: access_schema
    name: access.{{.Timestamp}}.parquet
    format: parquet

Performance Considerations

Batch Processing:

• Events are buffered until batch_size or max_size limits are reached
• Larger batches reduce S3 API calls and COPY INTO operations
• Balance batch size against latency requirements

Upload Optimization:

• Multipart uploads automatically handle large files
• Configure part_size for optimal network performance
• Default part size is AWS SDK default (5 MB)

COPY INTO Performance:

• COPY INTO commands are executed with configurable timeout
• Failed COPY operations return errors for retry logic
• Warehouse must be running (resumed) for COPY INTO to succeed

Warehouse State

Ensure the virtual warehouse is running before sending data. COPY INTO commands will fail if the warehouse is suspended. Configure warehouse auto-resume or manual resume procedures.

Error Handling

Upload Failures:

• Failed S3 uploads are retried based on sender configuration
• Permanent failures prevent COPY INTO execution
• Check S3 bucket permissions and network connectivity

COPY INTO Failures:

• Schema mismatches between files and tables cause failures
• Invalid SQL identifiers (database, schema, table names) are rejected at validation
• Check Snowflake query history for detailed error messages

Examples

Basic Configuration

Sending telemetry to Snowflake using S3 staging with Parquet format...

targets: - name: snowflake-warehouse type: amazonsnowflake properties: account: abc123.us-east-1 username: datastream_user password: "${SNOWFLAKE_PASSWORD}" database: PRODUCTION_DATA warehouse: COMPUTE_WH staging_bucket: datastream-staging region: us-east-1 table: EVENTS data_schema: event_schema name: events.{{.Timestamp}}.parquet format: parquet

With AWS Credentials

Using explicit AWS credentials for S3 staging access...

targets: - name: snowflake-secure type: amazonsnowflake properties: account: xyz789.us-west-2 username: security_user password: "${SNOWFLAKE_PASSWORD}" database: SECURITY_ANALYTICS warehouse: SECURITY_WH role: SECURITY_ADMIN staging_bucket: security-logs-staging staging_prefix: snowflake/ region: us-west-2 key: "${AWS_ACCESS_KEY}" secret: "${AWS_SECRET_KEY}" table: SECURITY_EVENTS data_schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet

Multi-Table Configuration

Routing different event types to separate Snowflake tables...

targets: - name: snowflake-multi-table type: amazonsnowflake properties: account: abc123.us-east-1 username: analytics_user password: "${SNOWFLAKE_PASSWORD}" database: ANALYTICS warehouse: ANALYTICS_WH staging_bucket: analytics-staging region: us-east-1 tables: - table: AUTHENTICATION_EVENTS schema: auth_schema name: auth.{{.Timestamp}}.parquet format: parquet - table: NETWORK_EVENTS schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet - table: APPLICATION_LOGS schema: app_schema name: app.{{.Timestamp}}.parquet format: parquet

High-Volume Configuration

Optimizing for high-volume ingestion with batch limits and compression...

targets: - name: snowflake-high-volume type: amazonsnowflake properties: account: abc123.us-east-1 username: streaming_user password: "${SNOWFLAKE_PASSWORD}" database: HIGH_VOLUME_DATA warehouse: LARGE_WH staging_bucket: streaming-staging region: us-east-1 batch_size: 100000 max_size: 134217728 part_size: 16 timeout: 600 table: STREAMING_EVENTS data_schema: streaming_schema name: stream.{{.Timestamp}}.parquet format: parquet compression: snappy

JSON Format

Using JSON format for flexible schema evolution and debugging...

targets: - name: snowflake-json type: amazonsnowflake properties: account: abc123.us-east-1 username: dev_user password: "${SNOWFLAKE_PASSWORD}" database: DEVELOPMENT warehouse: DEV_WH staging_bucket: dev-staging region: us-east-1 table: TEST_EVENTS name: test.{{.Timestamp}}.json format: json

With Normalization

Applying ECS normalization before loading to Snowflake...

targets: - name: snowflake-normalized type: amazonsnowflake properties: account: abc123.us-east-1 username: security_user password: "${SNOWFLAKE_PASSWORD}" database: SECURITY_DATA warehouse: SECURITY_WH staging_bucket: security-staging region: us-east-1 field_format: ECS table: ECS_EVENTS data_schema: ecs_schema name: ecs.{{.Timestamp}}.parquet format: parquet

Production Configuration

Production-ready configuration with performance tuning, AWS credentials, and multi-table routing...

targets: - name: snowflake-production type: amazonsnowflake properties: account: production.us-east-1 username: production_user password: "${SNOWFLAKE_PASSWORD}" database: PRODUCTION_ANALYTICS warehouse: PRODUCTION_WH role: DATA_ENGINEER staging_bucket: production-staging-bucket staging_prefix: datastream/snowflake/ region: us-east-1 key: "${AWS_ACCESS_KEY}" secret: "${AWS_SECRET_KEY}" batch_size: 50000 max_size: 67108864 part_size: 10 timeout: 300 field_format: ASIM tables: - table: SECURITY_EVENTS schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet compression: snappy - table: AUDIT_LOGS schema: audit_schema name: audit.{{.Timestamp}}.parquet format: parquet compression: snappy - table: NETWORK_FLOWS schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet compression: snappy

Troubleshooting

This section covers the errors you are most likely to see with the amazonsnowflake target, what causes each one, and how to fix it. Loading happens in two steps, and almost every failure belongs to one of them. Director first stages a file in the bucket named by staging_bucket, then runs a COPY INTO statement on Snowflake that reads that file back. The two steps use two different identities, so the step named in the error decides which credential or grant you have to fix.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission is missing?

The AWS credentials of the target, that is key and secret or the credentials the Director host resolves on its own, upload the staged file and nothing else. Snowflake reads that file with its own access to the bucket, because the COPY INTO command carries no storage integration and no credentials. Granting the Director credentials more rights on the bucket does not help the second step, and granting Snowflake more rights does not help the first one. Match the error against this table.

Error text	Missing right	Where it is granted
failed to upload to s3: ... api error AccessDenied: Access Denied	s3:PutObject, plus s3:AbortMultipartUpload for files larger than part_size	AWS, on arn:aws:s3:::my-bucket/snowflake-staging/*, for the identity the target uploads with
api error AccessDenied on a bucket whose default encryption is SSE-KMS	kms:GenerateDataKey and kms:Decrypt, typically	AWS, on the key that encrypts the bucket
Staged files accumulate under the prefix and are never removed	s3:DeleteObject	AWS, on the same prefix. The delete is best effort and its failure is not logged
snowflake API error (status 422) with code 091003	Read access to the staged objects for the Snowflake account itself	The bucket, on the staging prefix. This is not the identity in key and secret
snowflake API error (status 422) with code 002003	USAGE on the database and the schema, and INSERT on each table	Snowflake, to the user or to role
snowflake API error (status 422) with code 000606	USAGE on the warehouse, and either AUTO_RESUME or OPERATE so it can be resumed	Snowflake, on the warehouse in warehouse
snowflake API error (status 401) or (status 403) on every attempt	Valid username and password, and an account network policy that allows the Director host's egress address, typically	Snowflake, on the user and the account

"operation error STS: GetCallerIdentity" at startup

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: ..., api error InvalidClientTokenId: ...

Cause: the target checks its AWS identity before it accepts any data, and that call was refused. The code at the end names the problem. InvalidClientTokenId typically means the access key does not exist, SignatureDoesNotMatch that the secret is wrong or carries pasted whitespace, and ExpiredToken that session has expired. A reason ending in no EC2 IMDS role found means no credentials were found anywhere.

Fix: re-issue the key pair and paste both halves again. key and secret take effect only when both are set, so an unresolved ${AWS_ACCESS_KEY} reference makes the target fall back to the credential chain of the host, which then usually reports the IMDS message. Set both, or run Director on an instance whose role carries s3:PutObject on the staging prefix.

Nothing is sent while this lasts. Incoming data waits in the queue and is delivered once the target initializes, and no restart is needed.

"snowflake API error (status 401)" at startup

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: failed to test snowflake connection: snowflake API error (status 401): {"code":"390114","message":"Authentication token has expired"}

Cause: the target runs one SELECT 1 against the Snowflake SQL API at startup, and Snowflake refused the credentials. The status is what matters, because the body varies with the reason: a wrong user name or password typically reports a 390100 code, a locked or expired user another 3901xx code. A 403 instead typically means an account network policy does not allow the address Director connects from.

Fix: confirm username and password against the same account locator you put in account, and prefer a service user over a personal one. Accounts that require MFA or key-pair authentication for the SQL API typically refuse a password, and the target sends only username and password, so a user enrolled in MFA cannot be used here. If the reason mentions an unresolved reference instead, the ${SNOWFLAKE_PASSWORD} value is empty.

Nothing is sent while this lasts. A 401 or a 403 is retried for as long as it takes, and no data is dropped because of it.

"failed to upload to s3" with AccessDenied or another S3 code

[Error] [director] [target-<target id>] [snowflake-warehouse] Sender worker 0 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to upload to s3: operation error S3: PutObject, https response error StatusCode: 403, RequestID: ..., api error AccessDenied: Access Denied

Cause: the staged file could not be written, so the load never started. The operation named in the error is PutObject for small files and CreateMultipartUpload or UploadPart for files above part_size. The code after api error says why.

Code	What it means
AccessDenied	The identity has no s3:PutObject on the prefix, a bucket policy denies it, or the bucket is encrypted with a KMS key the identity cannot use
NoSuchBucket	staging_bucket is misspelled or the bucket is in another account. The target never creates buckets
PermanentRedirect	region is not the bucket's own region. The target does not follow the redirect for you
SlowDown or RequestTimeout	S3 is throttling the prefix, or the link is too slow for the object size
canceled, context deadline exceeded	The upload did not finish within timeout, which is 300 seconds by default

Fix: grant s3:PutObject and s3:AbortMultipartUpload on arn:aws:s3:::my-bucket/snowflake-staging/*, correct staging_bucket and region, and for the last two rows raise timeout, lower max_size, or raise part_size so fewer parts are sent. Adding {{.Year}}/{{.Month}}/{{.Day}} to the front of name spreads the keys over more prefixes when S3 throttles.

Nothing is lost here. The batch is discarded from memory, the payloads stay queued, and they are redelivered until the upload succeeds.

Snowflake code 091003, "Failure using stage area"

[Error] [director] [target-<target id>] [snowflake-warehouse] Sender worker 1 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: snowflake API error (status 422): {"code":"091003","message":"Failure using stage area. Cause: [Access Denied]"}

Cause: the file reached the bucket, and Snowflake cannot read it. The COPY INTO command names the object as a plain s3:// URL with no storage integration, no credentials and no named stage, so Snowflake has to be able to read that prefix on its own. A private staging bucket that only your AWS credentials can read produces this on every batch.

Fix: give the Snowflake account read access to the staging prefix on the bucket itself, then send again. No target setting passes a storage integration or credentials to the command, so a bucket that can only be read with them cannot be used for staging. Confirm the object is where you expect it first: it is written to s3://my-bucket/snowflake-staging/my_table/ unless you changed staging_prefix or name.

Nothing is lost. Each failed attempt deletes the staged object and the redelivery stages it again, so the batch is retried until Snowflake can read it.

Other Snowflake codes in "failed to execute COPY command"

The codes below arrive inside the same snowflake API error (status 422) wrapper and are retried until you fix them.

Code	What it means	Fix
002003	The database, schema or table does not exist, or the role cannot see it. Snowflake typically reports an object the role has no rights on as missing	Create the table, or grant USAGE on the database and schema and INSERT on the table. Identifiers are sent unquoted, so Snowflake typically folds them to upper case, and a table created with a quoted lower-case name cannot be targeted
000606	No warehouse is running the statement: warehouse is empty, or the warehouse is suspended with auto-resume disabled	Set warehouse and enable AUTO_RESUME, or grant OPERATE so it can be resumed
000630	A resource monitor has suspended the warehouse	Raise the quota or attach the target to another warehouse
000904	An identifier in the statement is not valid	Check database, schema and the table names for hyphens, dots and quotes
390114	The session credential is no longer accepted, typically after a password rotation	Update password
Status 429 or 5xx	A rate limit or an incident on the Snowflake side	Director keeps retrying. Larger batches through batch_size, max_size or a schedule lower the statement rate

"record rejected by target" after COPY INTO

[Error] [director] [target-<target id>] [snowflake-warehouse] Sender worker 0 execute() failed for snowflake-warehouse: failed to execute COPY command: record rejected by target: snowflake API error (status 422): {"code":"100038","message":"Numeric value 'lots' is not recognized","statementHandle":"01b2-0000"}

Cause: Snowflake read the staged file and refused its contents. Codes in the 100xxx range are data errors: 100038 is a value that does not fit the column type, 100069 is malformed JSON, and a NULL sent into a NOT NULL column lands here too. A status of 400 or 413 is treated the same way, and a 400 usually carries code 000904, a compilation error in the statement itself.

Fix: align the table definition with the file you are staging. For json, either load into a single VARIANT column or make the column names match the fields. Snowflake's COPY_HISTORY view shows the rejected rows and the exact reason for each one. If the file format is the problem rather than the data, read the next entry first.

This is one of three errors on this page that lose data. The others are a format value written in capitals, and a record routed to a table the target does not configure while drop_unknown_stream_events is false, described under The target is healthy but no rows arrive. The batch is redelivered at most four times and then dropped with deterministic failure for snowflake-warehouse after 4 attempts — dropping (giving up), and those records never reach my_table.

A format value in capital letters loses the batch

[Error] [director] [target-<target id>] [snowflake-warehouse] Sender worker 0 execute() failed for snowflake-warehouse: failed to execute COPY command: record rejected by target: snowflake API error (status 422): {"code":"100069","message":"..."}

Cause: format is validated without regard to case, but only the lower-case spellings are recognized when the file is written. PARQUET, Parquet and AVRO therefore pass validation, the file is staged as JSON, and the COPY INTO command still declares the format you asked for. Snowflake reads a JSON file as Parquet, refuses it, and the batch is dropped after four deliveries. Nothing in the configuration looks wrong, which makes this the most expensive mistake on this page.

Fix: write format in lower case, as json, avro or parquet, at the top level and in every tables[] entry. Two related traps produce the same rejection:

• csv, orc and xml pass validation and are also staged as JSON, so they fail in exactly the same way. Use one of the three supported values.
• A format inside a tables[] entry is not validated at all. Values such as jsonl, multijson and raw reach the statement as a format Snowflake does not accept, which typically comes back as a 400.

warning

Batches rejected this way are dropped after four deliveries. Correct the value before you resend the data, because a dropped batch cannot be replayed from Director.

"invalid schema name", "invalid table name" and other configuration errors

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: invalid schema name: event_schema.avsc

Cause: schema is the Snowflake schema, and it is validated as a SQL identifier. A file name such as event_schema.avsc is rejected and the target does not start. The same identifier rule applies to database and to every table name: letters, digits and underscores, not starting with a digit, and no quotes or dots.

Fix: use a bare identifier for schema, and put the catch-all table's Avro or Parquet schema in data_schema. schema is read as the file schema only when data_schema is empty, which is how a target written before data_schema existed still loads. Reference a library entry, a built-in model, inline JSON or a field list, not a path: a .avsc file name is not searched for.

These reach you through the same retry loop as a connection failure, and nothing is sent while any of them lasts.

Reason text	Fix
account is required for amazonsnowflake target	Set the property named in the message. username, password, database, staging_bucket and region report the same way
invalid database name: my-db	Hyphens, dots and quotes are not allowed. Use a bare identifier
invalid file format type: jsonl	The top-level format accepts json, avro and parquet. jsonl, multijson and raw are rejected here
at least one table must be configured (either 'table' or 'tables')	Set a catch-all table, or at least one entry under tables
invalid table configuration: table is required	Every entry under tables needs a table
invalid table configuration: name is required	Every entry under tables needs its own name. Only the catch-all has a default
invalid table configuration: data_schema is required for parquet format	The catch-all table is avro or parquet and neither data_schema nor schema is set. Set data_schema
invalid table configuration: tables[].schema is required for parquet format	Add schema to every tables entry whose format is avro or parquet, including entries that inherit that format from the top level
invalid default table schema: data_schema is not set, and the fallback schema does not hold an Avro/Parquet schema: …	data_schema is empty, so the Snowflake schema in schema (for example PUBLIC) was read as the catch-all table's file schema and could not be resolved. Set data_schema
invalid default table data_schema: …	The value in data_schema was not found in the schema library and is not valid inline JSON or a field list. The text after the colon says why
invalid schema format: invalid field format: security_schema	The value was not found in the schema library, so it was read as an inline field list and failed on the first token. Check the spelling and that the schema is deployed

A record the writer cannot encode fails the same way at send time, with the message coming from the format library rather than from Director. A single record that does not fit the Avro schema stops the whole stream behind it, so fix the schema or the pipeline rather than waiting it out.

"no such host", "context deadline exceeded", or certificate errors

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: failed to test snowflake connection: warehouse statement outcome indeterminate: failed to execute request: Post "https://my-account.snowflakecomputing.com/api/v2/statements": dial tcp: lookup my-account.snowflakecomputing.com: no such host

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: failed to test snowflake connection: warehouse statement outcome indeterminate: failed to execute request: Post "https://my-account.snowflakecomputing.com/api/v2/statements": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: the host name is built from account exactly as you typed it. no such host means that locator does not resolve, which is almost always a typo or a missing region part. context deadline exceeded means the connection was accepted or dropped silently and the startup probe gave up after 10 seconds, which points at a firewall or a proxy. The certificate message means a TLS-inspecting proxy is presenting its own certificate. A reason containing failed to parse response: invalid character '<' looking for beginning of value means something other than Snowflake answered, typically a proxy login page. Ignore the warehouse statement outcome indeterminate prefix on startup errors and read the text after failed to execute request:.

Fix: allow outbound HTTPS on port 443 from the Director host to:

• my-account.snowflakecomputing.com, built from account. A PrivateLink account typically needs the .privatelink suffix inside account itself
• sts.<region>.amazonaws.com, used once at startup for the identity check
• my-bucket.s3.<region>.amazonaws.com, for the staged files

HTTP_PROXY, HTTPS_PROXY and NO_PROXY are honored for both Snowflake and AWS, and instance metadata requests always go direct. For a private CA, install the CA certificate in the Director host's own trust store. The target reads no TLS options, so there is no setting to trust a custom CA or to skip verification.

Nothing is sent while this lasts, and it is retried until the host is reachable.

"still running: context deadline exceeded", and rows loaded twice

[Error] [director] [target-<target id>] [snowflake-warehouse] Sender worker 0 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: warehouse statement outcome indeterminate: statement 01b2-async-0001 still running: context deadline exceeded

Cause: Snowflake answers a long statement asynchronously after about 45 seconds, and Director then polls for the result until timeout, 300 seconds by default. This message means the statement had still not finished. An undersized warehouse, a very large staged file, or queued statements all cause it.

Fix: raise timeout, resize the warehouse, or lower max_size and batch_size so each load is smaller.

Treat this one carefully. The outcome is unknown, so the staged object is kept and the redelivery stages a second file for the same records. If the first statement completes in Snowflake afterwards, those rows are loaded twice. Check COPY_HISTORY after the target recovers, and clear the leftover objects from the staging prefix, or put a lifecycle rule on it. The same applies to a connection that drops mid-statement, which reports failed to poll statement instead.

The target is healthy but no rows arrive

Nothing fails in these cases, so there is no error to search for. Check them in order.

1. Events are dropped as unknown tables. With drop_unknown_stream_events at its default of true, an event whose destination table matches neither a tables[] entry nor the catch-all table is discarded without a log line. Only the target's dropped counter moves. Add the table, or set a catch-all table. Setting the flag to false turns the drop into a visible error that names the table, record rejected by target: destination "<table>" is not configured on the target, and the target has no catch-all. The record is still dropped after four attempts about 2 seconds apart, but alone, logged as Target "<name>" rejected record <n> of <payload> on its last delivery attempt — dropping that record and delivering the rest of the payload: <reason>, and the rest of the payload is delivered. The loss is the same as with the flag on, only later, so set the flag back once you have the name.

2. debug.dont_send_logs is enabled. Events are processed by the pipeline and then discarded before anything is buffered. Nothing is staged, nothing is loaded, no counter moves, and the target reports healthy. The only trace is a single line at startup, and only when debug.status is enabled as well. Remove the flag when you have finished testing.

3. The load succeeded but wrote no rows. The result of COPY INTO is not inspected, so a statement that returns success while loading zero rows is still counted as delivered. Snowflake typically skips a file it has already loaded, which is what you see after a retry has staged the same batch twice. COPY_HISTORY shows the rows loaded and the errors seen for each file.

4. A redelivered batch was skipped on purpose. After a partial success, a table that was already loaded is not staged again on the redelivery. This is correct de-duplication, but it looks like missing events when you compare counts.

5. Staged files pile up. The delete after a load is best effort and its failure is never logged, so a missing s3:DeleteObject leaves every staged file under the prefix. The data is loaded correctly. Grant the permission or add a lifecycle rule.

6. The files are not compressed. For json, only gzip is applied. Any other value, including the default zstd, stages an uncompressed file. Set compression: gzip. avro and parquet use their own codecs and are unaffected.

7. The upload used an unexpected identity. key and secret are used only when both are set. If one is empty, both are ignored and the host's credential chain is used, which can be another account.