Google BigQuery

Google Cloud Analytics

Synopsis

Creates a Google BigQuery target that streams data directly into BigQuery tables using the streaming insert API. Supports multiple tables, custom schemas, and field normalization.

Schema

- name: <string>
  description: <string>
  type: bigquery
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    dataset_id: <string>
    credentials: <string>
    batch_size: <numeric>
    timeout: <numeric>
    drop_unknown_stream_events: <boolean>
    ignore_unknown_values: <boolean>
    skip_invalid_rows: <boolean>
    max_bad_records: <numeric>
    field_format: <string>
    tables:
      - name: <string>
        schema: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y	-	Target name
description	N	-	Optional description
type	Y	-	Must be bigquery
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Google Cloud

Field	Required	Default	Description
project_id	Y	-	Google Cloud project ID
dataset_id	Y	-	BigQuery dataset ID
credentials	N	-	Service account credentials JSON (uses default credentials if not provided)

Tables

Field	Required	Default	Description
tables	Y	-	Array of target tables (at least one required). See Table Definitions below
tables[].name	Y	-	BigQuery table name
tables[].schema	N	-	Optional schema definition string. See Schema Format below

Streaming Options

Field	Required	Default	Description
drop_unknown_stream_events	N	true	Skip events for tables not defined in tables
batch_size	N	1000	Maximum number of rows per batch
timeout	N	30	Connection timeout in seconds
ignore_unknown_values	N	false	Accept rows with values that don't match the schema
skip_invalid_rows	N	false	Skip rows with errors and insert valid rows
max_bad_records	N	0	Maximum number of bad records allowed (0 = no limit)
field_format	N	-	Data normalization format. See applicable Normalization section

Table Definitions

You can define multiple tables to stream data into:

targets:
  - name: bigquery_multiple_tables
    type: bigquery
    properties:
      tables:
        - name: "security_logs"
          schema: "timestamp:TIMESTAMP,message:STRING,severity:STRING"
        - name: "system_logs"
          schema: "timestamp:TIMESTAMP,message:STRING,level:STRING"

Schema Format

The schema format follows the pattern: field1:type1,field2:type2,...

Supported types:

• STRING - Variable-length character data
• INTEGER or INT64 - 64-bit integer
• FLOAT or FLOAT64 - 64-bit floating point
• BOOLEAN or BOOL - True or false
• TIMESTAMP - Absolute point in time
• DATE - Calendar date
• TIME - Time of day
• DATETIME - Date and time
• BYTES - Binary data
• NUMERIC - Exact numeric value
• BIGNUMERIC - Larger numeric value
• GEOGRAPHY - Geographic data
• JSON - JSON data
• RECORD or STRUCT - Nested structure

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Google BigQuery target uses streaming inserts to send data in near real-time. Data is batched locally until batch_size is reached or when an explicit flush is triggered during finalization.

Each log event must specify its target table via the SystemS3 field — the value is matched against the name of an entry in tables. Events whose SystemS3 is empty or does not match any configured table are dropped if drop_unknown_stream_events is true (the default); otherwise they cause an error.

The target automatically parses JSON messages. If the message is not valid JSON, it creates a structured event with message and timestamp fields.

Authentication

The target supports two authentication methods:

1. Service Account JSON: Provide credentials directly in the configuration using credentials
2. Default Credentials: If credentials is not provided, the target uses Google Cloud's default credential chain (environment variables, gcloud CLI, GCE metadata service)

IAM Permissions

The service account requires the following IAM role:

IAM Role	Role ID	Purpose
BigQuery Data Editor	roles/bigquery.dataEditor	Insert rows into BigQuery tables via streaming API

Minimum permissions: bigquery.tables.updateData, bigquery.tables.get

Error Handling

The target provides flexible error handling:

• ignore_unknown_values: Allows inserting rows with extra fields not in the schema
• skip_invalid_rows: Continues inserting valid rows even if some rows fail
• max_bad_records: Limits the number of failed rows before returning an error

When skip_invalid_rows is enabled and errors occur, the target logs individual row errors when debug mode is enabled.

note

BigQuery streaming inserts have quotas and limits. Ensure your project has adequate quota for your ingestion rate.

Examples

Basic

Minimum configuration using default credentials:

targets:
  - name: basic_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "logs"
      tables:
        - name: "system_events"

With Credentials

Configuration with explicit service account credentials:

targets:
  - name: auth_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "logs"
      tables:
        - name: "application_logs"
      credentials: |
        {
          "type": "service_account",
          "project_id": "my-project",
          "private_key_id": "key-id",
          "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
          "client_email": "service-account@my-project.iam.gserviceaccount.com",
          "client_id": "123456789",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token"
        }

Multiple Tables

Configuration with multiple target tables and schemas:

targets:
  - name: multi_table_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "security_data"
      batch_size: 500
      tables:
        - name: "firewall_events"
          schema: "timestamp:TIMESTAMP,src_ip:STRING,dst_ip:STRING,action:STRING,bytes:INTEGER"
        - name: "authentication_events"
          schema: "timestamp:TIMESTAMP,username:STRING,success:BOOLEAN,source:STRING"
        - name: "dns_queries"
          schema: "timestamp:TIMESTAMP,query:STRING,response:STRING,client_ip:STRING"

High-Volume

Configuration optimized for high-volume streaming:

targets:
  - name: highvol_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "metrics"
      tables:
        - name: "performance_data"
      batch_size: 5000
      timeout: 60
      skip_invalid_rows: true
      max_bad_records: 100

With Error Handling

Configuration with flexible error handling:

targets:
  - name: flexible_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "logs"
      tables:
        - name: "app_logs"
      ignore_unknown_values: true
      skip_invalid_rows: true
      max_bad_records: 50

Normalized

Using field normalization for enhanced compatibility:

targets:
  - name: normalized_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "security"
      tables:
        - name: "normalized_events"
      field_format: "ecs"

With Debugging

Configuration with debug options for testing:

targets:
  - name: debug_bigquery
    type: bigquery
    properties:
      project_id: "my-project"
      dataset_id: "logs"
      tables:
        - name: "test_events"
      debug:
        status: true
        dont_send_logs: true

Environment Variables

Using environment variables for sensitive data:

targets:
  - name: secure_bigquery
    type: bigquery
    properties:
      project_id: "${GCP_PROJECT_ID}"
      dataset_id: "${BIGQUERY_DATASET}"
      tables:
        - name: "secure_logs"
      credentials: "${GCP_CREDENTIALS_JSON}"