Skip to main content

Keep

Synopsis​

Keeps only specified fields and removes all others.

Schema​

- keep:
    field: <string|string[]>
    schema: <string>
    schema_type: <string>
    requirement_filter: <string>
    root_field: <ident>
    description: <text>
    disabled: <boolean>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration​

The following fields are used to define the processor:

FieldRequiredDefaultDescription
fieldYField name(s) to keep. Accepts a single string, an array of strings, or glob patterns (*, ?, [])
schemaNSchema reference path. When set, the field list is loaded from the schema and field is ignored
schema_typeNparquetSchema file format. Accepted values: avro, parquet
requirement_filterNallFilters which schema requirement levels are loaded from the schema
root_fieldNWhen set, the keep operation runs inside this nested map instead of at the top level of the log entry
descriptionNExplanatory note
disabledNfalseDisable the processor without removing it from the pipeline
ifNCondition to run
ignore_failureNfalseContinue processing if field operations fail
ignore_missingNfalseContinue processing if the source field does not exist
on_failureNError handling processors
on_successNSuccess handling processors
tagNIdentifier

Details​

Removes all fields from the log entry except those explicitly specified. This processor performs the inverse operation of remove, allowing you to specify which fields to retain while discarding all others.

note

By default, the keep processor operates at the top level of the log entry. When specifying nested fields like user.id, the processor keeps the entire parent object (user) and not just the specific nested field. When root_field is set, the keep operation runs inside the specified nested map instead — only the keys within that map are filtered, and fields outside it are not affected.

The processor is useful for creating a minimal document with only essential fields, reducing the size of log entries, or ensuring consistent field presence for downstream systems.

When schema is set, the list of fields to keep is loaded from the referenced schema file and the field value is ignored. The schema_type field selects the schema format (avro or parquet, defaulting to parquet); requirement_filter controls which requirement levels from the schema are included, defaulting to all.

Field values support glob patterns: * matches any sequence of characters, ? matches any single character, and [...] matches a character set. Patterns are evaluated using filepath.Match semantics and apply to immediate key names within the target scope (top level or root_field map).

warning

Using the keep processor is a destructive operation that removes all non-specified fields. This can lead to loss of potentially useful data. Consider creating a copy of important data before applying keep operations or ensure that your field list is comprehensive.

Examples​

Single Field​

Keeping only essential identifier field...

{
  "id": "12345",
  "username": "jsmith",
  "password": "secret123",
  "email": "jsmith@example.com",
  "created_at": "2023-04-15T13:45:30Z"
}
- keep:
    field: "id"

removes all fields except id:

{
  "id": "12345"
}

Multiple Fields​

Preserving selected fields from a nested structure...

{
  "event": {
    "id": "evt-001",
    "type": "login",
    "timestamp": "2023-05-10T08:15:30Z",
    "debug_info": { "trace_id": "abc123", "session_data": "..." }
  },
  "user": {
    "id": "usr-123",
    "name": "John Smith",
    "role": "admin"
  },
  "context": {
    "ip": "192.168.1.1",
    "user_agent": "Mozilla/5.0...",
    "referrer": "https://example.com/login"
  }
}
- keep:
    field: 
      - "event.id"
      - "event.type"
      - "event.timestamp"
      - "user.id"
      - "user.role"

creates a minimal document with specific fields:

{
  "event": {
    "id": "evt-001",
    "type": "login",
    "timestamp": "2023-05-10T08:15:30Z"
  },
  "user": {
    "id": "usr-123",
    "role": "admin"
  }
}

Templates​

Dynamically selecting fields to keep...

{
  "event_type": "authentication",
  "authentication_username": "admin",
  "authentication_success": true,
  "authentication_method": "password",
  "source_ip": "10.0.0.1",
  "timestamp": "2023-06-20T14:25:16Z"
}
- keep:
    field: 
      - "event_type"
      - "{{event_type}}_username"
      - "{{event_type}}_success"
      - "source_ip"
      - "timestamp"

keeps event-specific fields based on event type:

{
  "event_type": "authentication",
  "authentication_username": "admin",
  "authentication_success": true,
  "source_ip": "10.0.0.1",
  "timestamp": "2023-06-20T14:25:16Z"
}

Conditional​

Applying different keep rules based on conditions...

{
  "log_level": "debug",
  "message": "User login attempt",
  "user_id": "u12345",
  "password_hash": "b5f...",
  "timestamp": "2023-07-15T09:12:45Z",
  "debug_data": {
    "trace": "...",
    "memory_usage": 1024,
    "stack": "..."
  }
}
- keep:
    if: "logEntry.log_level != 'debug'"
    field: 
      - "log_level"
      - "message"
      - "user_id"
      - "timestamp"
- keep:
    if: "logEntry.log_level == 'debug'"
    field: 
      - "log_level"
      - "message"
      - "user_id"
      - "timestamp"
      - "debug_data"

keeps different sets of fields based on log level