Microsoft Sentinel data lake

Microsoft Azure Security Lake

Synopsis

Creates a target that ingests log messages into Microsoft Sentinel data lake tables with lower ingestion costs and extended retention capabilities. Optimized for high-volume, high-fidelity log types like firewall logs, DNS logs, and network traffic requiring long-term storage.

tip

For more details on Microsoft Sentinel integration, refer to Microsoft Sentinel Overview and Microsoft Sentinel Integration. For Director Proxy deployment, see VirtualMetric Director Proxy.

Schema

- name: <string>
  description: <string>
  type: sentineldatalake
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    function_app: <string>
    function_token: <string>
    rule_id: <string>
    endpoint: <string>
    streams:
      - name: <string>
        rule_id: <string>
    stream: <string[]>
    buffer_size: <numeric>
    batch_size: <numeric>
    keep_phantom_fields: <boolean>
    drop_unknown_stream_events: <boolean>
    cache:
      timeout: <numeric>
    field_format: <string>
    detection_rules:
      - name: <string>
        query: <string>
        language: <string>
    unmatched_tables: <enum>
    smart_engine_mode: <enum>
    smart_engine_kept_fields:
      - table: <string>
        fields: <string[]>
    smart_engine_dropped_fields:
      - table: <string>
        fields: <string[]>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Core Settings

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be sentineldatalake
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Authentication

Field	Required	Default	Description
tenant_id	N*	-	Azure tenant ID (required for direct authentication)
client_id	N*	-	Azure client ID (required for direct authentication)
client_secret	N*	-	Client secret (required for direct authentication)
function_app	N*	-	Director Proxy endpoint URL (required for proxy forwarding)
function_token	N*	-	Director Proxy authentication token (required with function_app)

* = Conditionally required. Use either direct authentication (tenant_id, client_id, client_secret) OR Director Proxy forwarding (function_app, function_token), OR omit all credentials to use Managed Identity.

Managed Identity Authentication

Azure targets support Managed Identity authentication for credential-free access when Director is deployed on Azure infrastructure.

How it works: When tenant_id, client_id, and client_secret are omitted from the configuration, the target automatically uses Azure's DefaultAzureCredential, which attempts authentication in the following order:

1. Environment variables (AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET)
2. Workload Identity (for Kubernetes deployments)
3. Managed Identity (system-assigned or user-assigned)
4. Azure CLI credentials
5. Azure PowerShell credentials

Deployment requirement: Director must run on Azure infrastructure that supports Managed Identity:

• Azure Virtual Machines
• Azure App Service
• Azure Container Instances
• Azure Kubernetes Service (AKS)
• Azure Functions

Required permissions: The Managed Identity must be granted the appropriate Azure RBAC roles documented in each target's IAM Permissions section.

note

Managed Identity eliminates credential management overhead and is the recommended authentication method for Azure-hosted Director deployments.

Stream Configuration

Field	Required	Default	Description
endpoint	Y		DCE Address or DCE Resource ID
rule_id	N	-	Default Data Collection Rule (DCR) ID
streams	N	-	Array of stream configurations with name and optional rule_id
stream	N	-	Legacy string array of stream names
buffer_size	N	1048576	Encoding buffer size in bytes (1MB), applied per destination stream
batch_size	N	1000	Maximum messages per batch, counted per destination stream. Must be greater than 0
keep_phantom_fields	N	false	Keep fields not defined in DCR schema
drop_unknown_stream_events	N	true	Silently drop events for undefined streams. With false, such an event is rejected with an error, retried 3 times, then dropped
cache.timeout	N	300	Stream cache timeout in seconds. In autodiscovery mode, also how often the DCRs are read again. 0 or less turns that refresh off
field_format	N	-	Data normalization format. See applicable Normalization section

Detection Rules

A detection rule set assigned to this target turns its smart pipeline on: the target compiles the set with the Smart Engine processor and ships only the events those detections could need, field- and value-reduced.

In the web interface these entries are written for you — assign rules from Content Management > Detections, or from this target's own Smart Engine step and tab, and the assignment is regenerated on every configuration update. See Detections.

Field	Required	Default	Description
detection_rules	N	-	Detection rules to reduce against. Omit, or leave empty, to ship everything the pipelines produce
detection_rules[].name	Y		Rule name, used to report which detections matched
detection_rules[].query	Y		The rule's full source, inline. Not a reference to a rule stored elsewhere
detection_rules[].language	N	kql	Rule language: kql or yaral
unmatched_tables	N	keep	What to do with an event no assigned detection is scoped to: keep or drop
smart_engine_mode	N	active	What the Smart Engine does: off, learning or active. See Smart Engine Mode
smart_engine_kept_fields	N	-	Fields kept per table whatever the detections read: a list of table and fields. * is every table. Never keeps a log the rules drop
smart_engine_dropped_fields	N	-	Fields never sent per table, even when a detection reads them: a list of table and fields. * is every table. Required fields are never dropped

An entry missing either name or query is skipped, and the skipped count is reported in the debug log. If every entry is skipped the smart pipeline does not run at all, which ships everything rather than nothing — reduction fails open throughout.

unmatched_tables accepts keep and drop case-insensitively. Any other value is treated as keep and reported as a warning, so a typo is visible rather than a silent no-op.

warning

drop discards every event that no assigned detection is scoped to. A detection scoped to a table only applies to events that declare that table, so events with no declared table are never matched by a table-scoped rule and are governed entirely by this setting. A detection that fails open on a join, union or lookup also applies to the tables it reads that way, so their events are kept rather than dropped. Vet the assigned rule set against your traffic before switching it.

Kept and Dropped Fields

smart_engine_kept_fields and smart_engine_dropped_fields take the same shape, a list of tables and their fields:

smart_engine_kept_fields:
  - table: SecurityEvent
    fields: [Account, IpAddress]
  - table: "*"
    fields: [TenantId]
smart_engine_dropped_fields:
  - table: SecurityEvent
    fields: [EventData, Message]

Table and field names match case-insensitively. The table * means every table, and merges with a table's own list. The director passes the lists to the Smart Engine processor as kept_fields and dropped_fields.

A kept field is on every log of its table the engine keeps, whatever the rules read: a host name, a user, a correlation ID that an investigation or a workbook needs. It never decides whether a log is kept, value reduction never narrows it, and a field a log does not have is never added.

A dropped field is never sent to Microsoft Sentinel, even when a detection rule reads it. That rule then cannot work in Sentinel, and the console marks the table Non-compliant. A table's required fields, the time fields and a known table's schema-required columns, are never dropped. A dropped field wins over a kept one, and it never decides whether a log is kept.

Dropped fields work without any detection rule. With dropped fields and no rule, the engine still runs in smart_engine_mode: it keeps every log, whatever unmatched_tables says, and removes only the dropped fields. Learning mode predicts what would be removed; Active mode removes it.

note

Microsoft Sentinel ingests a kept field only if the table's Data Collection Rule defines it. The target strips any other column before sending, unless keep_phantom_fields is set.

Smart Engine Mode

smart_engine_mode is read case-insensitively:

Value	What the engine does
active	Drops every log no rule needs, and removes every field no rule reads and every dropped field.
learning	Evaluates every log as Active would, but drops and removes nothing. The Reduction tab shows what Active would remove.
off	Does not run. The rules and field choices stay saved, every log is sent unchanged, and the target still records what it sent.

The mode only applies to a target with detection rules or dropped fields; absent, it means active. Any other value is treated as active and reported as a warning. The web interface sets learning for a new target it assigns rules to.

Smart Engine Tab

In the web interface the target's Smart Engine tab writes detection_rules, unmatched_tables, smart_engine_mode, smart_engine_kept_fields and smart_engine_dropped_fields for you. From top to bottom it has three sections.

Smart Engine holds the settings: the mode, the assigned rules, Logs from uncovered tables, and Field choices, which counts the fields dropped and kept, for example "1 dropped · 2 kept". The mode tiles appear as soon as the target has a rule or a dropped field. With neither, the tab says the engine has nothing to run and offers Assign detection rules, which opens the editor with the rule picker over it; the mode is chosen afterwards.

Tables lists every Sentinel table the target sent logs to in the last 24 hours, every table the assigned rules read, and every table with a field choice. Each table has a status:

• Covered: a rule reads it.
• No coverage: no rule reads it.
• Non-compliant: a field a rule reads is dropped. The Rules column says how many rules break.

Above the list, four cards summarize the last 24 hours: Received, Sent to Sentinel (Would be sent in Learning mode), Read by no rule and Detection compliance. Savings available lists the biggest fields still sent from tables no rule covers, each with Drop and then Undo. Dropping one of these breaks no assigned rule.

Try it on your logs runs the engine over logs captured at the target. See Trying the Engine on Your Logs.

Choosing the Fields a Table Sends

Select a table under Tables to open its field editor. It lists every column of the table: its schema's columns and the fields its logs had.

• A ticked Keep box means the field is sent. Only a required field is locked.
• Reason says why a field is in its state: Required, Read by N rules, Breaks N rules, Kept by choice or Dropped by choice.
• Unticking a field a rule reads shows Non-compliant: 1 rule will not work, naming the field and the rule.

Search fields filters the list, Reset to default clears the table's choices, and a line under the list says how much more or less the change sends over the last 24 hours. Save changes writes the choices to smart_engine_kept_fields and smart_engine_dropped_fields; Discard changes drops them. While the rules above are being edited the editor is locked, with the reason shown.

In Learning mode the choices change the prediction on the Reduction tab, and with the mode Off they are saved; nothing is removed until the mode is Active.

Trying the Engine on Your Logs

Try it on your logs captures logs as they reach the target and runs the real engine over them with the rules, field choices and settings above. It drops nothing and saves nothing, and it runs the field choices alone on a target with no rule.

• Logs to capture offers 100, 250 or 500 logs. The capture stops at that count or after 60 seconds.
• In Off and Learning modes the logs are taken as they leave the target's pipelines, which is where the engine reads them. In Active mode they are taken before those pipelines, because the engine has already acted on what leaves them; the section says so.
• The capture is attributed to the origin smart-engine in the audit log.

The results show the same cards as the Reduction tab (logs, volume, and what the reduction is made of) with a Rules that matched card, then the same per-table breakdown as the Reduction tab and a per-rule breakdown. Inspect logs opens every captured log with its outcome and why: it matched a rule, no rule matched, or no rule reads its table. Each log lists its fields as kept, narrowed or removed.

While the rules are being edited, the same logs run again with the draft under With your unsaved changes, so the impact of a rule shows before it is saved.

Some logs cannot be read as the director reads them:

• A log with no Sentinel table is read as a log of an uncovered table, as the director does. A director older than the console names no table in a capture.
• A log over 256 KB, or text that is not a JSON object, cannot be simulated.
• One run takes at most 8 MB of logs. The section says how many logs were left out.

Reduction Tab

The Reduction tab shows what the engine removed, or in Learning mode would remove. A target with no engine (no rule and no dropped field), or with the mode Off, still records what it sent: the tab shows it as Observed: everything was sent unchanged, with its largest fields, before anything is turned on.

Largest fields and Most removed fields have Drop on each sent field no rule reads, and Keep on each field that is not sent. A field a rule reads shows Read by N rules instead, and a required field shows Required.

Scheduling

warning

Unlike most target types, sentineldatalake does not flush once per notification when left unconfigured. With neither interval nor cron set it flushes every 30 seconds.

See Scheduling and Pool Behavior for the interval, cron and max_items fields shared by all targets, and for how to override or opt out of this default.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Microsoft Sentinel data lake target provides cost-optimized ingestion for high-volume telemetry with extended retention requirements. Data lake ingestion offers significantly lower costs compared to standard DCR-based ingestion, making it ideal for firewall logs, DNS queries, network flows, and other high-fidelity telemetry requiring long-term storage.

Data Lake Benefits

Cost Efficiency - Data lake ingestion costs are substantially lower than standard analytics ingestion, enabling cost-effective processing of massive telemetry volumes that would be prohibitively expensive with traditional methods.

High Fidelity - Preserves complete log detail without sampling or field reduction, maintaining full forensic capability for security investigations and compliance auditing.

Extended Retention - Optimized for long-term storage of high-volume logs, supporting retention periods spanning months or years for compliance requirements and historical analysis.

Director Proxy Integration

The target supports two deployment models:

Direct Authentication - Director connects directly to Azure using service principal credentials (tenant_id, client_id, client_secret). This model requires Director to have network connectivity to Azure endpoints and credentials for the target subscription.

Director Proxy Forwarding - Director sends processed data to VirtualMetric Director Proxy (Azure Function) deployed in customer environment. Director Proxy uses Azure Managed Identity for credential-free access to Microsoft Sentinel data lake, eliminating the need to share Azure credentials with Director.

The Director Proxy model is particularly valuable for MSSP deployments where customers maintain complete control over Azure credentials while enabling centralized data processing and routing by the MSSP's Director infrastructure.

IAM Permissions

The Sentinel data lake target uses the same authentication and ingestion pattern as the standard Sentinel target. When using Service Principal or Managed Identity authentication, the following Azure roles are required:

Azure Role	Scope	Purpose
Monitoring Metrics Publisher	Data Collection Rule	Send data via DCR ingestion API
Monitoring Reader	Resource Group	List DCEs and DCRs for stream auto-discovery

The Monitoring Reader role can also be assigned per resource instead of on the resource group. Autodiscovery works with either model, and the same scopes apply to the managed identity of a Director Proxy. Never assign it at subscription scope:

Azure Role	Scope	Purpose
Monitoring Reader	Data Collection Endpoint	Read the DCE and find the DCRs linked to it
Monitoring Reader	Each Data Collection Rule	Read the DCR's stream and table declarations
Monitoring Metrics Publisher	Each Data Collection Rule	Send data via DCR ingestion API

A DCR that is missing a role is left out of discovery and only its tables fail. See the Troubleshooting section of the Microsoft Sentinel target for the resulting errors.

If the setup functions (DCE/DCR/table creation) are used, additional roles are required:

Azure Role	Scope	Purpose
Monitoring Contributor	Resource Group	Create and update DCEs and DCRs
Log Analytics Contributor	Log Analytics Workspace	Create and update custom log tables

Stream Discovery

When endpoint is specified as a Resource ID (not HTTPS URL), the target automatically discovers available Data Collection Rules and their associated streams. This autodiscovery feature simplifies configuration by eliminating manual stream enumeration.

While the target runs, Director reads the DCRs again every cache.timeout seconds (default 300), in the background, and applies your own streams list to the result as it does at startup. Sends never wait for that read: each worker picks up the new set of streams at its next send.

• A stream that a new or newly readable DCR adds is used from the next send.
• A stream whose DCR, endpoint, table or columns changed, or that is no longer discovered, is switched or dropped only after the records already buffered for it have been delivered. A refresh never discards a buffered record.
• A refresh that fails, or that finds no streams, logs a Warning and keeps the streams from the last successful discovery.

The refresh is off when cache.timeout is 0 or less, or when endpoint is an https:// URL, which turns autodiscovery off.

Stream configurations can be filtered using the streams array to limit ingestion to specific tables. Each stream configuration supports independent DCR IDs via the rule_id field, enabling flexible routing to different data collection rules.

Field Management

The target automatically detects table schemas and validates incoming data against defined columns. When keep_phantom_fields is false (default), fields not defined in the target schema are automatically removed before ingestion, preventing schema validation errors.

warning

Disabling keep_phantom_fields removes undefined fields. Ensure all required fields are included in your DCR schema.

Data is buffered per destination stream until batch_size records or buffer_size bytes are reached for that stream, or the flush schedule fires. Whichever limit binds first triggers the upload; the record that would have exceeded it rides the next one rather than being dropped. Because both limits are per stream, a target fanning out to several streams accumulates up to that much for each of them before a flush. The drop_unknown_stream_events setting (default: true) silently discards events for streams not configured in the target, preventing processing failures for unexpected data types.

With drop_unknown_stream_events: false, an event that maps to no configured stream is rejected instead, and logged as record rejected by target: no DCR stream for table "<table>" (stream "<stream>" is not configured on the target), or as no DCR stream for definition <N> for an event with no table. The target stays up and does not reinitialize. The payload is retried about every 2 seconds, and on its 4th delivery it is dropped (dropping (giving up)) and its undelivered events are added to the target's dropped counter. Events before the rejected one in the same batch are delivered, and events after it are dropped with it. So false does not hold data until the stream is fixed: it makes the drop visible, then drops the data all the same.

warning

Enabling drop_unknown_stream_events silently discards unmatched events. Monitor data flow to ensure expected streams are properly configured.

Field Normalization

The field_format property normalizes log data to standard formats before ingestion:

• csl - Common Security Log format
• asim - Advanced Security Information Model

Normalization ensures consistent field naming and structure across diverse log sources, improving query efficiency and security analytics capabilities.

Examples

Basic Configuration

Minimum configuration using direct Azure authentication:

targets:
  - name: sentinel_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Managed Identity

Using Managed Identity for credential-free authentication on Azure infrastructure:

targets:
  - name: managed_identity_data_lake
    type: sentineldatalake
    properties:
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Director Proxy

Configuration using Director Proxy for credential-free forwarding:

targets:
  - name: proxy_data_lake
    type: sentineldatalake
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Filtered Streams

Configuration with specific stream filtering and custom settings:

targets:
  - name: filtered_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-FirewallLogs"
        - name: "Custom-DNSLogs"
      keep_phantom_fields: false
      drop_unknown_stream_events: true
      cache:
        timeout: 600

High-Volume Processing

Optimized configuration for high-volume log ingestion:

targets:
  - name: high_volume_data_lake
    type: sentineldatalake
    pipelines:
      - normalization
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      batch_size: 5000
      field_format: "asim"
      streams:
        - name: "Custom-FirewallLogs"
          rule_id: "dcr-00000000000000000000000000000000"
        - name: "Custom-DNSLogs"
          rule_id: "dcr-11111111111111111111111111111111"

Debug Configuration

Testing configuration with debug enabled:

targets:
  - name: debug_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      debug:
        status: true
        dont_send_logs: true  # Test mode - doesn't actually upload

Troubleshooting

The sentineldatalake target authenticates, discovers streams, and uploads in exactly the same way as the sentinel target, so it reports the same errors for the same causes: missing Monitoring Reader or Monitoring Metrics Publisher roles, credential errors, an empty stream configuration, rejected uploads, and Director Proxy errors. Use the Troubleshooting section of the Microsoft Sentinel target for the full list of errors, causes, and fixes, and Target Delivery Errors for how Director logs and retries target failures.

One difference applies when reading that section for this target: log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples. The errors in the Director Proxy table apply unchanged; the target wizard sets function_app and function_token through the same Proxy field as Microsoft Sentinel's.