Normalize

Transform

Synopsis

Converts log field names between different normalization formats (ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM, GELF, OTel).

Schema

- normalize:
    source_format: <string>
    target_format: <string>
    field: <ident>
    ignore_unmapped_fields: <boolean>
    ignore_rules: <boolean>
    use_json_parser: <boolean>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
`source_format`	N	auto-detect	Source format (`ecs`, `cim`, `asim`, `cef`, `leef`, `csl`, `ocsf`, `udm`, `gelf`, `otel`). `otel` is never auto-detected; see below
`target_format`	Y	-	Target format (`ecs`, `cim`, `asim`, `cef`, `leef`, `csl`, `ocsf`, `udm`, `gelf`, `otel`)
`field`	N	-	Nested field containing the data to normalize
`ignore_unmapped_fields`	N	`false`, or `true` for `csl`	Drop fields with no mapping in the target format. Unset, it is `false` everywhere except a `csl` target, where it defaults to `true`
`ignore_rules`	N	`false`	Skip schema enforcement for the target format. See below for which targets enforce
`use_table`	N	`false`	Write the normalized result into a virtual table rather than the map
`source_table`	N	`false`	Read the source from the entry's selected virtual table instead of the map, and drop that table afterwards. With no table selected the source is the map, so map-mode records are unaffected
`select_table`	N	`true`	`use_table` only. Whether this normalize also claims the marshal source. Set `false` for an intermediate normalize whose table is not the terminal one
`additional_extensions`	N	`true`	CSL target only. Whether fields with no CSL mapping are folded into the `AdditionalExtensions` string. Set `false` when the destination is ASIM only, where that string is never emitted
`use_json_parser`	N	`false`	CSL target only. Write unmapped fields as structured keys under `AdditionalExtensions.<name>` instead of leaving them at the root. It does not parse the source field as JSON
`description`	N	-	Documentation note
`if`	N	-	Conditional expression
`ignore_failure`	N	`false`	Skip processing errors
`on_failure`	N	-	Error handling processors
`on_success`	N	-	Success handling processors
`tag`	N	-	Identifier for logging
`disabled`	N	`false`	When `true`, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

The processor automatically detects source formats when not specified and handles field name transformations while preserving values. Format detection is case-insensitive.

Supported Formats

Format	Description
`ecs`	Elastic Common Schema
`cim`	Splunk Common Information Model
`asim`	Microsoft Sentinel ASIM
`cef`	Common Event Format
`leef`	Log Event Extended Format
`csl`	Microsoft Sentinel CommonSecurityLog
`ocsf`	Open Cybersecurity Schema Framework
`udm`	Google SecOps Unified Data Model
`gelf`	Graylog Extended Log Format
`otel`	OpenTelemetry semantic conventions, as a flat log record

otel has no characteristic field to detect it by. To normalize a record that is already OTel-shaped, set source_format: otel explicitly; it is inferred only when an earlier normalize step in the same pipeline already produced an otel record.

Schema Enforcement

Schema enforcement runs by default for six target formats: asim, csl, ocsf, udm, gelf and otel. The rules normalize timestamps, validate event types, normalize security actions and bring field values into line with the target schema. Set ignore_rules: true to skip it.

For otel, the pass converts the two typed log-record fields:

severity_number is derived from severity_text when that holds a known severity word. A syslog severity code, carried over from log.syslog.severity.code, is the fallback. severity_text wins when both are present.
time_unix_nano is converted from the @timestamp it was renamed from into Unix nanoseconds. A timestamp that cannot be parsed keeps its original value rather than becoming 0, so a bad input stays visible.

The other targets — ecs, cef and leef — map fields and apply no enforcement pass, so ignore_rules has no effect on them.

cef and leef do not produce CEF or LEEF records

The processor renames fields; it does not write a wire format. With target_format: cef or leef, the result is still a JSON document whose keys follow CEF or LEEF naming. It has no CEF:0|… or LEEF:…| pipe header, which is what ArcSight and QRadar parsers match on, so those products do not parse it as CEF or LEEF.

To deliver real CEF or LEEF, use the content pack that renders the record before the target sends it. gelf is different: its enforcement pass adds the fields GELF requires, so the result is a valid GELF document.

Field Mappings

The processor automatically detects the source format based on characteristic fields. Common mappings between formats:

	ECS	CIM	ASIM	UDM
Network
	`source.ip`	`src`	`SrcIp`	`principal.ip`
	`destination.ip`	`dest`	`DstIp`	`target.ip`
	`network.direction`	`direction`	`NetworkDirection`	`network.direction`
Event
	`@timestamp`	`_time`	`TimeGenerated`	`metadata.event_timestamp`
	`event.action`	`action`	`EventType`	`metadata.product_event_type`
	`event.severity`	`severity`	`EventSeverity`	`security_result.severity`
User
	`source.user.name`	`user`	`ActorUsername`	`principal.user.userid`
	`source.user.id`	`user_id`	`ActorUserId`	`principal.user.windows_sid`
	`source.user.email`	`user_email`	`ActorUserEmail`	`principal.user.email_addresses`

See the Appendix sections ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM, GELF and OTel for the full field mappings.

warning

Field mapping is non-reversible if the targeted format doesn't have equivalent fields. Test the conversions beforehand with sample data.

Examples

ECS to CIM

ECS fields...

{
  "source": {
    "ip": "128.232.110.120"
  },
  "destination": {
    "ip": "192.168.1.1"
  },
  "network": {
    "direction": "inbound"
  }
}

- normalize:
    source_format: ecs
    target_format: cim

are mapped to CIM fields:

{
  "src": "128.232.110.120",
  "dest": "192.168.1.1",
  "direction": "inbound"
}

CIM to ECS

CIM fields...

{
  "src": "128.232.110.120",
  "dest": "192.168.1.1",
  "direction": "outbound"
}

- normalize:
    source_format: cim
    target_format: ecs

are mapped to ECS fields:

{
  "source": {
    "ip": "128.232.110.120"
  },
  "destination": {
    "ip": "192.168.1.1"
  },
  "network": {
    "direction": "outbound"
  }
}

Auto-detection

Auto-detection discovers CIM...

{
  "_time": "2023-01-01T00:00:00Z",
  "src": "128.232.110.120",
  "dest": "192.168.1.1",
  "direction": "outbound"
}

- normalize:
    target_format: asim

and maps the fields to ASIM:

{
  "TimeGenerated": "2023-01-01T00:00:00Z",
  "SrcIp": "128.232.110.120",
  "DstIp": "192.168.1.1",
  "NetworkDirection": "outbound"
}

Error Handling

Handling conversion errors...	`{ "source": { "invalid": true } }` `- normalize: target_format: cim on_failure: - set: field: error value: "Conversion failed"`
captures the error information:	`{ "source": { "invalid": true }, "error": "Conversion failed" }`

ECS to UDM

ECS network event fields...

{
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "source": {
    "ip": "192.168.1.100",
    "port": 54321,
    "user": { "name": "jdoe" }
  },
  "destination": {
    "ip": "10.0.0.50",
    "port": 443
  },
  "event": {
    "action": "connection",
    "severity": "low"
  }
}

- normalize:
    source_format: ecs
    target_format: udm

are mapped to Google SecOps UDM fields:

{
  "metadata": {
    "event_timestamp": 1705315800000000,
    "event_type": "NETWORK_CONNECTION"
  },
  "principal": {
    "ip": "192.168.1.100",
    "port": 54321,
    "user": { "userid": "jdoe" }
  },
  "target": {
    "ip": "10.0.0.50",
    "port": 443
  },
  "security_result": {
    "severity": "LOW",
    "action": "UNKNOWN_ACTION"
  }
}

Synopsis​

Schema​

Configuration​

Details​

Supported Formats​

Schema Enforcement​

Field Mappings​

Examples​

ECS to CIM​

CIM to ECS​

Auto-detection​

Error Handling​

ECS to UDM​