Smart Engine

Analytics Flow Control Security

Synopsis

Registers your KQL detection rules and forwards only what those rules could ever need. An event that no detection could match is dropped; an event that survives keeps only the fields the applicable detections read or return.

The detections are your existing SIEM rules, used unchanged. The engine does not convert them into a pipeline — for that, see From KQL.

note

This processor can also be configured indirectly. Detection rules stored under Detections and assigned to a Microsoft Sentinel target are deployed as that target's detection_rules property, and the target compiles them into a smart_engine processor for you. The two routes drive the same processor — this page documents the pipeline-level configuration, and the target pages document the assignment.

Schema

- smart_engine:
    detections:
      - name: <string>
        query: <string>
    library: <string[]>
    select: <object>
    table_field: <ident>
    unmatched_tables: <enum>
    mark_only: <boolean>
    mark_field: <ident>
    value_reduction: <boolean>
    field_reduction: <boolean>
    critical_fields: <string[]>
    kept_fields: <map>
    dropped_fields: <map>
    description: <text>
    if: <script>
    tag: <string>
    disabled: <boolean>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>

Configuration

One of detections, library or dropped_fields must be set. With none of them, the processor fails — it does not drop.

Field	Required	Default	Description
`detections`	Y*		Detection rules given inline. Each entry is a `name` and a `query`, where `query` is raw KQL taken verbatim from the SIEM rule.
`library`	Y*		Names of Microsoft Sentinel analytics-rule packs assigned through the platform Library. Globs are accepted, so `"*"` selects every assigned pack. Only each rule's KQL is used. Unioned with `detections`.
`select`	N	all rules	Narrows which rules from `library` are registered. See below.
`table_field`	N	`_vmetric.table`	The event field holding the table name a detection is scoped against.
`unmatched_tables`	N	`keep`	What to do with an event that no detection is scoped to: `keep` or `drop`.
`mark_only`	N	`false`	Dry run. Marks each event with the decision the engine would have taken, but never drops or reduces anything.
`mark_field`	N	`_smart`	Root field the markers are written under.
`value_reduction`	N	`true`	Strip individual values from comma-separated fields.
`field_reduction`	N	`true`	Strip fields no applicable detection needs.
`critical_fields`	N		Fields kept on every kept event, whatever the detections read. Names only: they never rescue an event from being dropped, and an absent field is not created.
`kept_fields`	N		`critical_fields` scoped by table: a map from a table name to the fields kept on every kept event of that table, whatever the detections read. The key `"*"` applies to every table and merges with a table's own list. Tables match case-insensitively, and value reduction never narrows a kept field. Like `critical_fields`, it never rescues an event from being dropped and never creates an absent field.
`dropped_fields`	Y*		A map from a table name to the fields removed from every forwarded event of that table — matched, whole-row, fail-open or kept uncovered — whatever the detections read or `kept_fields` keeps. The key `"*"` applies to every table and merges with a table's own list. Tables match case-insensitively. A table's required fields, the time fields and a known table's schema-required columns, are never removed. It never decides whether an event is kept: with `dropped_fields` and no detection, every event is kept and only these fields are removed.
`description`	N		Explanatory note.
`if`	N		Condition that must be true for the processor to run.
`tag`	N		Identifier for this processor instance.
`disabled`	N	`false`	When `true`, the processor is skipped.
`ignore_failure`	N	`false`	Continue pipeline processing if the processor fails. This does not mean "keep filtering and ignore errors" — see the warning below.
`on_failure`	N		See Handling Failures.
`on_success`	N		See Handling Success.

* = One of detections, library or dropped_fields is required.

warning

ignore_failure: true silently turns this processor off. The drop is signalled to the pipeline as an error value, and the ignore_failure check runs before that value is inspected — so a matched event is kept instead of dropped, no error is logged, and the pipeline reports success. The processor appears to run normally while filtering nothing.

Use it only if you genuinely want a pass-through. To tolerate real errors without losing the filtering, leave ignore_failure unset and handle the failure with on_failure instead.

Rule Packs

Rules are supplied as packs assigned through the platform Library, one file per pack:

Rule type	Library path
Sigma	`library/sigma/<name>.yml`
Sentinel	`library/sentinel/<name>.yml`
YARA	`library/yara/<name>.yar`

A pack's name is its filename without the extension, and that name is what library refers to.

Globs are accepted, so "*" selects every assigned pack of that type and "win_*" selects those whose names begin with win_. A glob that matches nothing contributes no rules and raises no error. A name given exactly, however, must be assigned — if it is not, the processor fails with an error naming it.

The Sentinel and Smart Engine processors read the same Sentinel rule packs, so a pack assigned once can be used by both.

Select

All criteria are combined with AND; the values within one criterion are combined with OR. Matching is case-insensitive.

Field	Description
`tables`	Rules whose declared data types include any of these, such as `SecurityEvent` or `CommonSecurityLog`.
`connectors`	Rules by required data connector id.
`severities`	`informational`, `low`, `medium`, or `high`.
`kinds`	`Scheduled` or `NRT`.
`tactics`	MITRE tactic. Separator-insensitive, so `CommandAndControl` and `command-and-control` are equivalent.
`techniques`	MITRE technique id. A parent id also selects its sub-techniques, so `T1059` selects `T1059.001`.
`ids`	Rule id, matched exactly.
`name_prefix`, `name_contains`, `name_regex`	Rule title matching.
`desc_contains`	Rule description matching.
`tags`	Rule tags.

Details

warning

This processor runs on the Director only. An Agent skips it as if it were disabled and passes its records on unchanged; the Director logs a warning when it sends that Agent its configuration. Put it in a pipeline the Director runs, such as a route pipeline or a target postprocessing pipeline.

How an Event Is Decided

For each event, in order:

The event's table is read from table_field.
The applicable detections are selected: those scoped to that table, plus any scoped to no table at all. A detection's table is the source table named in its KQL; a query that starts from a datatable or externaldata let takes the table it joins in. A detection that fails open also applies to every other table it reads through join, union or lookup. A query that starts with union and has no leading table, or one that reads through an ASIM parser function such as _Im_NetworkSession(...), has no table and applies to every event.
Each applicable detection's condition is evaluated against the event. The condition is the detection's pre-aggregation where chain.
The event is dropped only when at least one detection was applicable and none of them matched.
If no detection was applicable, unmatched_tables decides. The default is to keep.

An event kept because no detection applied to it is forwarded whole — there is no mask to apply — except that the dropped_fields of its table are still removed. Detection-based reduction only happens to events that some detection actually matched.

What Survives on a Kept Event

The surviving fields are the union of:

Every field read by the conditions of all applicable detections — matched or not. This is deliberate. A detection that did not match may still test a field for absence, and stripping that field could make the rule fire at the SIEM against an event that should never have triggered it.
Every field returned by the matched detections, meaning their project, project-keep, distinct, and summarize columns.
The critical_fields.
The kept_fields of the event's table.
The marker, table, and system fields.

Everything else is removed. A field in the dropped_fields of the event's table never survives, even when a detection reads it or kept_fields keeps it, unless it is one of the table's required fields.

Masking operates on top-level fields. A detection reading Properties.CommandLine preserves the whole Properties object, not just that one subfield. The engine over-keeps rather than under-keeps.

Fail Open

If the engine cannot understand a detection, that detection matches every event it is scoped to, and reduction is switched off for those events. It never discards an event because it failed to understand a rule.

A detection fails open when its query uses a construct the condition compiler does not handle — a join, union or lookup with another table, mv-expand, parse, evaluate and the like — when the query cannot be fully parsed, or when a where tests a let that reads an ingested table (a toscalar(), tabular or function let over a table).

The consequence is worth stating directly: a rule the engine cannot parse produces no savings. And if such a rule also has no resolvable source table, it applies to every event in the pipeline, which switches reduction off entirely — one such rule in a large pack is enough to make the processor a no-op.

The engine warns when this happens, naming the detection and saying that it disables reduction globally until it is scoped to a table or narrowed with a project or summarize. If you register a large pack with library: ["*"] and see no volume reduction, that warning identifies the rule responsible.

Approximated Detections

A test inside a where that the engine cannot evaluate per event does not fail the whole detection open. That test is left out of the condition, and the rest of the filter still reduces. A detection compiled this way is approximate.

Construct in a detection	What the engine does
A `where` testing a column computed by an earlier `extend`	Leaves that test out and keeps the rest of the filter
A `where` testing a `let` that reads no ingested table: a constant expression, a `datatable`, an `externaldata` feed, or a Sentinel watchlist (`_GetWatchlist`)	Leaves that test out
A `where` calling a function the engine has no equivalent for, such as `extract()`	Leaves that test out
`let x = pack_array("a", "b")`	Reads it like `dynamic(["a", "b"])`
A join of a table with itself: the same table again, a tabular `let` over it, or an `as` alias of the query's own rows	Keeps an event when either side's filters keep it

A detection that leaves a test out and also reads another ingested table still fails open, so that table's rows stay protected.

Leaving a test out only ever keeps more. The test counts as true, or as false where it sits under a not, so the detection keeps every event it needs and some it does not. Savings shrink; detections are never lost.
The fields stay. The fields a left-out test reads stay on the event, and value reduction never trims them, so the SIEM still has what it needs to run the test itself.
The sentinel processor is unchanged. It treats an approximate detection as unresolved, like a fail-open one, and stamps nothing unless enrich_on_unresolved: true.
The coverage view reports an approximate detection as ok, without the Fails open tag. The API carries approximate: true and names the left-out tests in reason.

Time Windows and Thresholds

Time predicates such as ago(), between, and now() are treated as true. The event has just arrived, so it is inside any recency window by definition; the SIEM re-checks the window at query time. The time field itself is still preserved, so that check can be made.

A threshold applied after a summarize — | summarize count() by User | where count_ > 5 — is ignored per event. Every contributing event is kept, even one that could never on its own reach the threshold, because the SIEM cannot compute the aggregate without them. Reduction from such a rule comes from its pre-aggregation where, not from the threshold.

Whole-Row Detections

A detection with no project, summarize, distinct, or project-keep presents the entire row at the SIEM, so nothing may be stripped from an event it matched. The same holds for a detection that outputs every column some other way: a wildcard project *, or a summarize aggregate over * — arg_max(x, *), arg_min(x, *), take_any(*) — which returns the whole row it picks. The common Sentinel idiom summarize arg_max(TimeGenerated, *) by … is one of these, so expect no field reduction from such rules. A matched whole-row detection therefore disables field reduction for that event.

Value reduction is disabled more readily: any applicable whole-row or fail-open detection disables it, whether or not it matched.

Value Reduction

For a string field that contains commas, and that some applicable detection tests with a positive literal membership test — ==, =~, has, has_any, in, and their relatives — the value is split on commas and a part is kept when a tested term appears in it as a whole word, matched case-insensitively. Kept parts are rejoined, preserving the original spacing.

Whole-word matching is what the SIEM itself applies when it re-runs the detection downstream, so a part carrying extra tokens around the term survives: a test for sshd keeps sshd (pid 1), not only a part that is exactly sshd.

Value reduction is blocked for any field that an applicable detection returns as a result column, or reads in any way other than a positive literal membership test: a regular expression, contains, startswith, a relational comparison, a negation, an absence test, or as an argument to a function.

Set value_reduction: false to keep field values untouched and reduce fields only.

Markers

Field	Written	Description
`_smart.rules`	On every kept event	The names of the matched detections, in configuration order. An empty list when the event was kept by the `unmatched_tables` policy.
`_smart.keep`	Only under `mark_only`	The decision the engine would have taken.

Both are written under mark_field. Nothing is written to a dropped event.

Rollout

Deploy with mark_only: true first. Every event is marked with the decision the engine would have made and nothing is dropped or stripped, so the volume of events carrying _smart.keep: false tells you the reduction to expect, and the distribution of _smart.rules tells you which detections are carrying it. Set mark_only: false to enforce.

Examples

Reducing a Matched Event

Registering a detection that reads two fields and returns two others...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "50126",
  "UserPrincipalName": "alice@example.com",
  "IPAddress": "10.0.0.7",
  "UserAgent": "Mozilla/5.0",
  "CorrelationId": "abc-123"
}

- smart_engine:
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

The detection matches, so the event is kept and reduced to the fields the rule reads and returns. UserAgent and CorrelationId are needed by nothing and are removed...

{
  "_vmetric": { "table": "SigninLogs" },
  "_smart": { "rules": ["failed-signin"] },
  "ResultType": "50126",
  "UserPrincipalName": "alice@example.com",
  "IPAddress": "10.0.0.7"
}

Dropping an Event No Detection Wants

The same detection, against a successful sign-in...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

- smart_engine:
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

A detection was applicable to this table and it did not match, so the event is dropped. Nothing is forwarded...

// dropped

Keeping Fields the Detections Do Not Read

Registering an assigned Sentinel pack, narrowed to one table, and guaranteeing two fields the rules may not mention...	`- smart_engine: library: - "*" select: tables: - SecurityEvent critical_fields: - TimeGenerated - Computer`
Every assigned pack's `SecurityEvent` rules are registered. `TimeGenerated` and `Computer` survive on every kept event even when no rule reads them — but they never rescue an event from being dropped...	`{ "_smart": { "rules": ["Suspicious process executed"] }, "TimeGenerated": "2026-07-14T09:12:00Z", "Computer": "WS-014", "Process": "powershell.exe" }`

Measuring Before Enforcing

Running the same configuration as a dry run...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

- smart_engine:
    mark_only: true
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

The event would have been dropped, so it is marked and forwarded intact. Counting events with _smart.keep set to false gives the reduction to expect before anything is discarded...

{
  "_vmetric": { "table": "SigninLogs" },
  "_smart": { "rules": [], "keep": false },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

Synopsis​

Schema​

Configuration​

Rule Packs​

Select​

Details​

How an Event Is Decided​

What Survives on a Kept Event​

Fail Open​

Approximated Detections​

Time Windows and Thresholds​

Whole-Row Detections​

Value Reduction​

Markers​

Rollout​

Examples​

Reducing a Matched Event​

Dropping an Event No Detection Wants​

Keeping Fields the Detections Do Not Read​

Measuring Before Enforcing​