Linux Datasets

Linux datasets define collection rules for Linux devices (dataset type linux). Most collector types, including file log collection, are configured through the dataset creation wizard — see Management; the connection event and script execution collectors below are configured via device YAML. For the dataset and profile model, see Overview.

Collector Types

• Linux System Events (linux_host_log_collector): File path input for the system log source.
• Linux Audit Events (linux_audit_report_log_collector): File path input for the audit log source.
• Linux Firewall Events (linux_firewall_log_collector): File path input for the firewall log source.
• Linux TCP Connections (linux_tcp_connection_collector): Established TCP connections with process attribution and destination geolocation.
• Linux Remote Connections (linux_remote_connection_collector): Remote login sessions (SSH and console) from host session records.
• Linux Script Execution (linux_execute_collector): Runs a script or executable and collects its standard output as log events.

Connection Event Collectors

The TCP connection (linux_tcp_connection_collector) and remote connection (linux_remote_connection_collector) collectors capture host network activity as ECS events. The TCP collector reports established TCP connections with process attribution and destination geolocation. The remote connection collector reports remote login sessions (SSH, derived from pts* devices) and local console sessions (tty*) from the host session records. Both accept only a pipeline_name property for routing; collection cadence, connection caching, and country-level geolocation are handled automatically.

note

These collectors are configured via device YAML files under config/devices/ — they are not available in the dataset creation wizard.

Enabling TCP and remote connection collection on a Linux device, routing both to a pipeline...	devices: - id: 123458 name: linux-host type: linux status: true definitions: - name: linux_tcp_connection_collector status: true properties: pipeline_name: my-pipeline - name: linux_remote_connection_collector status: true properties: pipeline_name: my-pipeline
Each established TCP connection is emitted as an ECS event with process and destination geo...	{ "@timestamp": "2026-07-24T10:15:03.472Z", "event": { "dataset": "linux.tcp_connection", "category": ["network"], "type": ["connection", "start"], "action": "established" }, "host": { "name": "linux-host", "ip": ["10.0.0.12"] }, "network": { "transport": "tcp", "type": "ipv4" }, "source": { "ip": "10.0.0.12", "port": 52344 }, "destination": { "ip": "140.82.112.3", "port": 443, "geo": { "country_iso_code": "US", "country_name": "United States" } }, "process": { "name": "curl", "pid": 4821 } }
A remote login session is emitted with the authenticated user and session type...	{ "@timestamp": "2026-07-24T10:12:55.000Z", "event": { "dataset": "linux.remote_connection", "category": ["authentication", "session"], "type": ["start"], "action": "SSH" }, "message": "admin@10.0.0.5", "host": { "name": "linux-host", "ip": ["10.0.0.12"] }, "network": { "transport": "tcp" }, "user": { "name": "admin" }, "source": { "ip": "10.0.0.5", "geo": { "country_iso_code": "US", "country_name": "United States" } } }

Script Execution

The linux_execute_collector runs a script or executable on the Linux host and collects its standard output as log events. Each execution is defined as an input with its own interpreter, arguments, schedule, and pipeline routing, so a single dataset can run multiple scripts.

note

This collector is configured via device YAML files under config/devices/. Scripts must be placed in the agent's user/scripts (operator-owned, survives upgrades) or package/scripts (product-shipped) directory; the path property is a file name only.

Input properties

Each entry under inputs: defines one script execution. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
path	Y	string	—	Script or executable file name (not a path). Resolved under the agent's scripts directories. Absolute paths and .. traversal are rejected.
interpreter	N	string	—	Interpreter that runs the script. Linux: sh, bash, or exe. Windows: ps1, cmd, or exe. exe runs path directly. There is no implicit default; if unset (or unrecognized), shell is used instead.
shell	N	string	—	Custom interpreter for the fall-through case (e.g. python3 or an executable path). On Windows with interpreter: ps1, selects the PowerShell edition: pwsh (7+), powershell (5.1), or unset (auto).
args	N	list	—	Arguments passed to the script or executable.
environment	N	map	—	KEY: VALUE pairs appended to the inherited process environment.
working_directory	N	string	inherit	Working directory for the child process.
timeout	N	int	120	Maximum run time in seconds. A non-positive value falls back to 120. On timeout, the entire child process group is terminated.
max_bytes	N	int	0	Cap on captured output in bytes. 0 means unbounded.
emit_on_error	N	boolean	false	When true, a non-zero exit code still emits the captured stdout. A timeout or a failure to start never emits, regardless of this setting.
execution_policy_bypass	N	boolean	false	Windows ps1 only. Adds -ExecutionPolicy Bypass to the PowerShell invocation.
cron	N	string	—	Per-input schedule (cron expression). An empty value or "0" runs the script on every collection cycle.
max_retries	N	int	1	Number of times a failed run is retried on subsequent collection cycles.
skippable	N	boolean	true	When true, missed schedule slots collapse into a single run. When false, each missed slot is replayed.
pipeline_name	N	string	—	Route the output to a specific preprocessing pipeline by name.
ignore_time	N	boolean	true	Skip time-based filtering on the emitted events.

Output

Captured stdout is converted to log events one line at a time:

• Trailing line terminators are trimmed once; if nothing remains, no events are emitted (an empty or whitespace-only run produces nothing).
• The remaining output is split on newlines, producing one event per line. A trailing carriage return on each line is stripped, so CRLF output is handled.
• Interior blank lines are preserved; only trailing blank lines from a final newline are dropped.
• stderr is never emitted as events. The exit code is logged and gates emit_on_error.

Example

Running a health-check shell script every five minutes and routing its output to a pipeline...	devices: - id: 123459 name: linux-host type: linux status: true definitions: - name: linux_execute_collector status: true inputs: - id: 3001 name: Disk Health status: true properties: path: disk-health.sh interpreter: bash args: - "--all" cron: "*/5 * * * *" timeout: 30 pipeline_name: my-pipeline
Each line of standard output becomes a separate log event...	{ "message": "/dev/sda1 usage 62%" } { "message": "/dev/sdb1 usage 41%" }

File Log Collection

The linux_file_log_collector collects lines from arbitrary log files on Linux hosts. It supports glob path expansion, lookback-based backfill, multiline parsing, include/exclude filtering, character-set decoding, and per-pipeline routing. The path syntax uses Linux-style paths (e.g. /var/log/myapp/*.log).

note

File log datasets can be created through the dataset creation wizard (Linux File Logs) or configured via device YAML files under config/devices/ using the schema below.

Device-level property

Field	Required	Type	Default	Description
file_log_concurrency	N	int	1	Maximum number of inputs processed in parallel per device.

Input properties

Each entry under inputs: defines one file log source. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
path	Y	string	—	File path or glob pattern. Supports wildcards (e.g. /var/log/myapp/*.log, C:\Logs\*\app-*.log).
start_date	N	int	300	Lookback window in seconds. 0 collects the last second only; a negative value disables time-based filtering.
ignore_cache	N	boolean	false	Skip the persisted file-position cache (re-read from the start).
ignore_old_date	N	boolean	false	Skip old-date filtering.
ignore_retention	N	boolean	false	Skip retention filtering.
ignore_time	N	boolean	false	Skip time-based filtering.
date_format	N	string	—	Custom timestamp layout using Java-style tokens (e.g. yyyy-MM-dd HH:mm:ss). Used to extract an event timestamp from each line.
line_parser	N	object | string	—	Multiline detection rules. See Line parser.
filter_mode	N	string	exclude	include or exclude. Synonyms: inclusive/allow (include), exclusive/deny (exclude).
filter_rules	N	array	—	Include/exclude rules. See Filter rules. Alias: filters.
encoding	N	string | int	—	Character encoding for the file. Accepts an alias (e.g. utf-8, windows-1252) or a numeric decoder ID. See Encoding aliases.
pipeline_name	N	string	—	Route matched lines to a specific preprocessing pipeline by name.

Line parser

line_parser controls how multi-line log entries are reassembled. It accepts either an object or a bare string shorthand.

Accepted type values:

Value	Aliases	Behavior
regex	1	A regex pattern detects where a new entry begins.
newline	new_line, 2	Each raw line is a separate entry.
string	prefix, 3	Lines beginning with a literal string mark a new entry.

Object fields:

Field	Required	Type	Description
type	N	string | int	One of the values above. If omitted and regex is set, regex is assumed.
regex	N	string	Pattern used by regex, or the literal prefix used by string/prefix.
value	N	string	Alias for regex.
date_based	N	boolean	Merge continuation lines using date boundaries detected via date_format.
has_space	N	boolean	Treat leading whitespace on a line as a continuation of the previous entry.

A bare string at line_parser: is treated as type: regex with that pattern:

line_parser: '^\d{4}-\d{2}-\d{2}'

Filter rules

filter_rules is an array of include/exclude rules applied after line parsing. The effect of each match (keep or drop) is determined by filter_mode.

Each rule accepts:

Field	Required	Type	Description
type	N	string | int	regex (1) or string (2). If omitted, inferred from which field is set.
regex	N	string	Regex pattern to match against the line.
source	N	string	Wildcard pattern to match against the line.
value	N	string	Alias for source.

A bare array of regex strings is also accepted and is equivalent to type: regex for each entry.

Encoding aliases

encoding accepts a numeric decoder ID or one of the following aliases. Dashes, underscores, dots, and spaces are ignored during alias lookup, so utf-8, utf_8, and UTF 8 all resolve identically.

Alias(es)	ID
utf8	1
utf8bom	2
utf16be	3
utf16le	4
utf16bebom	5
utf16lebom	6
gbk	11
latin1, iso88591	15
windows1250, cp1250	50
windows1251, cp1251	51
windows1252, cp1252	52
windows1256, cp1256	56

Example

Collecting a rotating application log on Linux with regex-based multiline detection and an include filter for errors and warnings...

devices: - id: 123457 name: app-linux type: linux status: true properties: file_log_concurrency: 2 definitions: - name: linux_file_log_collector status: true inputs: - id: 1001 name: Application Logs status: true properties: path: /var/log/myapp/*.log start_date: 300 date_format: yyyy-MM-dd HH:mm:ss line_parser: type: regex regex: '^\d{4}-\d{2}-\d{2}' date_based: true filter_mode: include filter_rules: - type: regex regex: '^(ERROR|WARN)' encoding: utf-8 pipeline_name: my-pipeline