File

Long Term Storage

Synopsis

Creates a file target that writes log messages to files in various formats like JSON, MultiJSON, Avro, Parquet, with support for various compression methods and schemas.

Schema

- name: <string>
  description: <string>
  type: file
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    location: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    field_format: <string>
    batch_size: <integer>
    max_size: <integer>
    max_rows_per_rowgroup: <integer>
    buffer_size: <integer>
    data_page_version: <string>
    metadata: <key-value>
    locations: <location[]>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be file
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Files

Files can have the following properties:

Field	Required	Default	Description
location	N	<service-root>	File output directory
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	File name template
format	N	"json"	File format. See Formats below
compression	N	zstd	Compression algorithm. See Compression below
extension	N	Matches format	Custom file extension
schema	N*	-	Data schema for Avro / Parquet formats, and required for both of them. Can be a Library schema name, a built-in schema name, a path to a schema file, or an inline JSON definition
batch_size	N	100000	Maximum number of messages per file
max_size	N	32MB	Maximum file size before rotating
field_format	N	-	Data normalization format. See applicable Normalization section

Multiple Locations

You can define multiple output locations with different settings:

targets:
  - name: multi_location_logs
    type: file
    properties:
      locations:
        - id: "security_logs"
          path: "/var/log/security"
          schema: "CommonSecurityLog"
          format: "parquet"
        - id: "system_logs"
          path: "/var/log/system"
          format: "json"

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The file target supports writing to multiple file locations with different formats and schemas. When using SystemS3 field in your logs, the value will be used to route the message to the location with a matching ID.

A schema is required for the Avro and Parquet formats, and there is no default. Without one the target does not start, and Director reports invalid location configuration: schema is required for avro format, naming the format you set.

The target supports the following built-in schema templates:

• Syslog - Standard schema for Syslog messages
• CommonSecurityLog - Schema compatible with Common Security Log Format (CSL)

You can also reference custom schema files by name (without the .json extension). The system will search for schema files in:

1. User schema directory: <user-path>/schemas/
2. Package schema directory: <package-path>/schemas/

Schema files are searched recursively in these directories, and filename matching is case-insensitive.

note

Files with no messages (i.e. with counter=0) are automatically removed when the target is disposed.

Templates

The following template variables can be used in the file name:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	file
{{.Table}}	Location ID	security_logs
{{.Thread}}	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	3
{{.ServiceRoot}}	The service root directory	/opt/vmetric

Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
jsonl	The same JSON lines output as json, written with the jsonl extension
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema
raw	A line format, written one record per line

note

A format name that is not recognized falls back to JSON lines, and your spelling is kept as the file extension. No error is reported, so check the extension when the output is not what you configured.

Compression

Files can use the following compression algorithms:

Format	Default	Compression Codecs
JSON, JSONL, MultiJSON, Raw	None	gzip
Avro	zstd	deflate, snappy, zstd
Parquet	zstd	gzip, snappy, zstd, brotli, lz4

The compression property defaults to zstd, which the line formats do not accept, so JSON, JSONL, MultiJSON and Raw files are written uncompressed unless you set compression: gzip. A codec a format does not accept is ignored without an error. Avro and Parquet also accept zstandard as a spelling of zstd.

Parquet Options

These properties apply when format is parquet. Set them on the target, or on an entry in locations.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Rows buffered in memory before a row group is closed
buffer_size	N	262144	Page buffer size, in bytes
data_page_version	N	V2	Data page version. V1 is also accepted
metadata	N	-	Key/value pairs written to the file footer

note

max_size is measured against the file on disk, so rows still held in memory do not count towards it. A Parquet file can overshoot max_size by up to one row group.

Examples

JSON

Configuration for a JSON output (as "json" is the default format, no need to specify it):

targets:
  - name: json_logs
    type: file
    properties:
      location: "/var/log/vmetric"

Multiple Locations

Configuration for multiple output locations with different formats:

targets:
  - name: multi_location_logs
    type: file
    properties:
      locations:
        - id: "security"
          path: "/var/log/vmetric/security"
          format: "parquet"
          schema: "CommonSecurityLog"
          compression: "zstd"
        - id: "system"
          path: "/var/log/vmetric/system"
          format: "json"
        - id: "application"
          path: "/var/log/vmetric/app"
          format: "multijson"
          name: "app_{{.Year}}_{{.Month}}_{{.Day}}.json"

Avro with Built-in Schema

Configuration for an Avro output with compression using a built-in schema:

targets:
  - name: syslog_avro
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "avro"
      compression: "snappy"
      schema: "Syslog"

Avro with Custom Schema File

Configuration for an Avro output using a custom schema file:

targets:
  - name: custom_avro
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "avro"
      compression: "zstd"
      schema: "MyCustomSchema"

This will look for MyCustomSchema.json in the schema directories.

Avro with Inline Schema

Configuration for an Avro output with an inline schema definition:

targets:
  - name: avro_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "avro"
      compression: "zstd"
      schema: |
        {
          "type": "record",
          "name": "Log",
          "fields": [
            {"name": "epoch", "type": "long"},
            {"name": "message", "type": "string"}
          ]
        }

Parquet with Built-in Schema

Configuration for a Parquet output with compression using a built-in schema:

targets:
  - name: csl_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "parquet"
      schema: "CommonSecurityLog"
      compression: "brotli"

Parquet with Custom Schema File

Configuration for a Parquet output using a custom schema file:

targets:
  - name: custom_parquet
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "parquet"
      compression: "zstd"
      schema: "NetworkTrafficSchema"

Parquet with Inline Schema

Configuration for a Parquet output with an inline schema definition:

targets:
  - name: parquet_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "parquet"
      compression: "zstd"
      schema: |
        {
          "timestamp": {
            "type": "INT64",
            "logicalType": "TIMESTAMP_MILLIS"
          },
          "message": {
            "type": "STRING",
            "compression": "ZSTD"
          },
          "level": {
            "type": "STRING"
          }
        }

Kusto Schema Conversion

You can also use Kusto schema format, which will be automatically converted:

targets:
  - name: kusto_format
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "parquet"
      schema: "timestamp:datetime,message:string,level:string,source:string"

Windows

Configuration for a Windows environment with a proper path structure:

targets:
  - name: windows_logs
    type: file
    properties:
      location: "C:\\ProgramData\\VMetric\\Logs"
      format: "json"
      name: "windows_{{.Year}}\\{{.Month}}\\system_logs.json"

Daily Rotation with Templates

Configuration with daily file rotation using template variables:

targets:
  - name: daily_logs
    type: file
    properties:
      location: "/var/log/vmetric"
      format: "avro"
      compression: "zstd"
      name: "logs_{{.Year}}_{{.Month}}_{{.Day}}.avro"
      schema: "Syslog"

Troubleshooting

This section covers the errors you are most likely to see with the file target, what causes each one, and how to fix it. The target opens no network connection. Its connection is the local filesystem, so every failure here is a path, permission, schema, or disk problem on the machine where Director runs.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

note

No failure on this target is classified as a permanent rejection, so nothing is dropped after a fixed number of attempts. A failed batch is retried until the cause is fixed, and the data waits in the Director queue meanwhile. See Persistent Storage for how long queued data is kept.

What the path needs

Grant the following to the account the Director service runs as, on the directory named by location, or by path in each entry of locations. A directory you can write to from your own terminal is often not writable by the service.

Requirement	When it applies	Error when it is missing
Permission to create the directory, granted on its parent	Only when the directory does not exist yet. Director creates it at startup	output path ... is not accessible: mkdir ...
Permission to create and write files in the directory	Always. Each batch opens its own file	open ... followed by the operating system's access error
Permission to traverse every component of the path, plus share access for a UNC or mounted path	Always	mkdir ... or open ... followed by the operating system's not-found error
Free space and quota headroom on the volume	Always, on every write and every flush	The operating system's out-of-space error on write ... or sync ...
A schema value that resolves	Only with format: avro or format: parquet	invalid location configuration: schema is required for avro format, or invalid schema format: ...
Permission to delete files in the directory	Optional. Files that end up with no records are removed when they are closed	None. Empty files are left behind

"output path ... is not accessible"

[Error] [director] [target-<target id>] [json_logs] Failed to reinitialize target "json_logs" (attempt 1). Reason: output path "/var/log/vmetric" for target "json_logs" is not accessible: mkdir /var/log/vmetric: <operating system error>

Cause: the directory does not exist and Director could not create it. The wording after the last colon comes from the operating system, and it tells you which case you have.

Typical wording	What it means
permission denied	The service account cannot create a directory under the parent. Linux wording
read-only file system	The volume is mounted read-only. Linux wording
Access is denied.	The service account has no Modify permission on the parent. Windows wording
The system cannot find the path specified.	The drive letter or the parent directory does not exist on this machine. Windows wording

Fix: create the directory and grant the Director service account write access to it, or set location to a directory that is already writable. The next retry picks it up, so no restart is needed.

Data impact: the target never starts, so nothing is written. Incoming data waits in the queue until the path works.

note

This check runs only on paths that contain no template variable. If location or name includes something like {{.Year}}, the directory is created at write time instead, and the same problem appears later as Sender worker 1 ThreadSafeInit on reinit failed or as a failed delivery.

"invalid schema format" or "schema is required for ... format"

[Error] [director] [target-<target id>] [csl_logs] Failed to reinitialize target "csl_logs" (attempt 2). Reason: invalid schema format: invalid field format: CommonSecurtyLog

Cause: schema did not resolve to anything Director recognizes. Both avro and parquet require a schema, and there is no default. When the value is missing altogether, the reason reads invalid location configuration: schema is required for avro format, naming the format you set.

A schema name is matched, case-insensitively, against library schemas, the built-in schema names and, for Parquet, schema files in the schema directories. When none of them matches, the value is read as a Kusto column list, and a misspelled name fails there. So invalid field format: followed by your own schema name almost always means the name is spelled wrong, as in the sample above.

Fix: correct the name. Syslog and CommonSecurityLog are the built-in schemas. For a schema file, give the file name without the .json extension. For a Kusto column list, use name:type pairs separated by commas.

Two related messages point at the body of the schema rather than at its name:

• failed to parse Avro schema: ... means an inline or library Avro schema was rejected. The text after the last colon names the offending type or field.
• failed to parse Parquet schema: ... means a Parquet field definition is not valid, for example invalid bit width 24 for INT type; must be 8, 16, 32, or 64.

Data impact: the target never starts, so nothing is written. Incoming data waits in the queue.

"no space left on device", or the disk fills up

[Error] [director] [target-<target id>] [json_logs] Sender worker 1 execute() failed for <payload>: target broken: write /var/log/vmetric/vmetric.1789171200123456789.json: <operating system error>

Cause: the volume is full, or a quota is exhausted. The wording is the operating system's own. It is typically no space left on device on Linux, and typically There is not enough space on the disk. on Windows. The same wording appears on sync ... rather than write ... when the failure lands at rotation, and the line then reads Sender worker 1 Finalize failed on flush for target "json_logs": ....

Writes are buffered, so the error surfaces when the buffer is flushed, not on the record that filled the volume.

Fix: free space or raise the quota. Director never removes the files it has written, so plan for the growth. batch_size and max_size bound the size of each file, not the total. Archive or prune the directory with your own scheduled job, or move location to a volume with room.

Data impact: nothing is lost. The batch is retried until it is written. A partial file can be left in the directory, and a parquet file left this way has no footer, so it typically cannot be read. Remove those leftovers before archiving.

A file cannot be renamed or deleted while it is open

/var/log/vmetric/vmetric.1789171200123456789.json

Symptom: an archiving or rotation job cannot move or delete the newest file in the output directory, and reports that the file is in use.

Cause: Director holds the current output file open while it writes to it. On Windows, an open handle typically blocks another process from renaming or deleting that file until the handle is released. The same job typically succeeds on Linux, where those operations are allowed while a file is open.

The handle is released when the file is closed, which happens when batch_size or max_size is reached, at each interval or cron tick, and after 30 seconds with no data in scheduled mode. Without interval or cron the target runs in immediate mode, where each delivered payload gets its own file that is closed straight away.

Fix: have the job skip the most recent file, or select only files whose timestamp in the name is older than the current one. Setting interval or cron also bounds how long any one file stays open.

The reverse case looks like a permission problem. If another process holds a file open at the path the target wants to write, the open typically fails on Windows with Access is denied., and the target reports it as a worker initialization failure.

The same file keeps growing, or "failed to create Avro encoder"

[Error] [director] [target-<target id>] [daily_logs] Sender worker 1 execute() failed for <payload>: target broken: failed to create Avro encoder: invalid avro file

Cause: the resolved file name repeats between batches, so each batch reopens the previous file and appends to it. Two configurations do that.

• A name with no {{.Timestamp}}. A date-only name such as logs_{{.Year}}_{{.Month}}_{{.Day}}.avro resolves to one file per day by design. For JSON lines that is usually what you want. For avro, the target has to read the existing file's header before it can append, and the message above is what you see when that file is not a valid Avro container, for instance because another tool or another format wrote it. For parquet, which carries a single footer at the end of the file, appending to an existing file typically produces a file that readers cannot open.
• A misspelled template variable. A typo such as {{.Timestmap}} is not reported. The literal text becomes part of the name, files appear as vmetric.{{.Timestmap}}.json, and every batch appends to that one file. Check names against the Templates table.

The same root cause has a third face. When two or more writers resolve to one fixed path, initialization fails with failed to generate unique file path after 10 attempts for thread 1.

Fix: include {{.Timestamp}} in name for avro and parquet, so each batch gets a fresh file. Give every target and every entry in locations a distinct name, or add {{.TargetName}} or {{.Thread}} to it. Move or delete any foreign file already sitting at the resolved path.

Data impact: batches are retried until the path is usable. Nothing is lost, but an Avro or Parquet file that was appended to can be unreadable.

The target looks healthy but no file appears

Check these in order.

1. debug.dont_send_logs is enabled. Records are processed and accepted, nothing is written, and the empty file opened for the batch is removed when it is closed. The target's Events Out counter does not move either, so traffic on the pipeline with a flat counter is the signature. With debug.status also enabled, Director logs Log sending is disabled for this target (json_logs). Logs will be processed by the pipeline but will not be sent to the target. Set debug.dont_send_logs: false.

2. location is not set. The files are written to the Director installation directory, next to the service itself. Set location explicitly.

3. A Windows path is configured on a Linux host. A value such as C:\ProgramData\VMetric\Logs becomes the relative directory C:/ProgramData/VMetric/Logs under the service working directory, and is created there without an error. Use a POSIX path on Linux hosts.

4. The files are there under a name you did not expect. See the entry above. A misspelled template variable is kept in the file name as you typed it.

Two more mismatches are accepted silently. They change the output rather than stopping it.

• An unrecognized format falls back to JSON lines but keeps your spelling as the file extension. A file carrying an extension you did not intend, holding JSON lines, means the format name was not recognized. Use json, jsonl, multijson, avro, parquet, or raw.
• An unrecognized compression is ignored and the file is written uncompressed. Each format accepts its own codecs, listed under Compression. The default is affected too. With format: json and no compression set, the default zstd is not a JSON codec, so the file is not compressed. Set compression: gzip for JSON output that has to be compressed.

One record blocks the batch

[Error] [director] [target-<target id>] [syslog_avro] Sender worker 1 execute() failed for <payload>: target broken: avro: unknown enum symbol: WARN

Cause: one value in the batch cannot be encoded against the schema, for example a value that is not among the symbols the schema's enum declares. The batch is not accepted, and since nothing here counts as a permanent rejection, the same batch is retried indefinitely. A single record stops the whole payload.

Fix: widen the schema to accept the value, or normalize the field with a pipeline processor before it reaches the target.

Too many small files

Cause: with no interval or cron, the target runs in immediate mode and writes one file per delivered payload. This is the default, and it is not a fault.

Fix: set interval or cron so that one file covers a whole tick. See Scheduling and Pool Behavior. batch_size and max_size still rotate a file early when it reaches either limit.