Example: A Syslog-To-JSON Data Stream

This section will help you configure and run a full, end-to-end data flow using devices, targets, and a pipeline, walking you through a common use case.

Scenario

We will create a pipeline that ingests logs from a Syslog source, processes and categorizes them, and forwards them to a JSON file for storage and analysis.

We will implement the following scenario:

• Ingest data - Listen to Syslog to receive messages.

• Extract information - Extract specific information from the Syslog messages and map them to fields that will be stored in a JSON file.

• Write the information - Write the fields mapped from the extracted information to a JSON file.

We need to work in an environment that has access to Syslog messages. We also need write permissions to the directory where the JSON file will be stored, although since we will be using our working directory that is taken care of.

We assume by now you understand the underlying logic of a pipeline since you have reviewed the previous example.

Setup and Trial

Loading include…

For convenience, we will place all our configurations in a single file named syslog-to-json.yml which we will again place in our working directory.

Configure the device

First, we have to define a device that will receive the log data. We will use the configuration in our ingesting data example since that will adequately serve our needs:

syslog-to-json.yml

devices:
  - id: 1
    name: from_syslog
    type: syslog
    status: true
    properties:
      protocol: udp
      address: 127.0.0.1
      port: 514

This configuration will create a UDP Syslog server listening on port 514.

note

On some operating systems, binding to port 514 may require elevated privileges. If you encounter permission issues, either run with appropriate privileges or use a higher unprivileged port (e.g. 5514) and adjust your sender accordingly. ::

Put this device configuration in our file.

Configure the target

Next, we will define a target for the JSON file storage:

syslog-to-json.yml

targets:
  - name: to_json
    type: file
    status: true
    properties:
      location: C:\Users\<username>\Documents\Logs
      name: syslog-event-data.json
      compression: zstd
      format: "json"

Note that we have specified a certain path for location.

caution

You should create the folder you want to use for location in advance, and make sure that you have write permissions to it.

With this configuration, we are setting up a JSON file target that will store the processed Syslog data. We have also enabled ZSTD compression for efficient storage.

Put this target configuration in our file.

Configure the pipeline

We are now ready to configure our pipeline:

syslog-to-json.yml

pipelines:
  - name: extract_user_event
    processors:
      - grok:
          field: message
          patterns:
            - "%{DATA}: %{GREEDYDATA:raw_message} - %{NUMBER}"
      - set:
          field: Event
          copy_from: raw_message
      - remove:
          field: message
          ignore_missing: true
      - remove:
          field: raw_message
          ignore_missing: true

This configuration will

• using the grok processor, parse the Syslog header information to read the field raw_message

• write the raw_message value to the Event field in the JSON file

• remove the message and raw_message fields of Syslog from the output

note

To avoid raising an exception, we chose to ignore missing fields.

Put this pipeline configuration in our file.

Configure the route

Finally, we have to create a route to connect our device to the pipeline and then to our target. Put the following definition in our configuration file:

syslog-to-json.yml

routes:
  - name: syslog_to_json
    devices:
      - name: from_syslog
    pipelines:
      - name: extract_user_event
    targets:
      - name: to_json

This configuration will

• connect to the Syslog server and receive data from it,
• apply the extract_user_event pipeline to the ingested data, and
• write the processed data to the JSON file specified.

Put this route configuration in our file.

Run the data stream

Let's put it all together: we have defined a device, a target, and a pipeline, and specified a route that will link the device to the pipeline and the target. At this point we have everything ready.

We can now start running our configuration. First, invoke Director's -console switch to monitor the status messages:

PowerShell

.\vmetric-director -console

Bash

./vmetric-director -console

We will once again use our own tool to send our messages to Syslog. Switch to the other terminal, type the following, and press Enter:

PowerShell

.\vmetric-director -generator -now -mode syslog -count 1 -address "127.0.0.1:514" -message "John Doe says 'Hello world'"

Bash

./vmetric-director -generator -now -mode syslog -count 1 -address "127.0.0.1:514" -message "John Doe says 'Hello world'"

Loading include…

As Director starts sending the messages, the screen should look like this:

1 messages sent successfully in 0 ms with 0 error(s) at 2025-06-29 22:44:28.5619632 +0300 +03 m=+0.054953001
1 messages sent successfully since 2025-06-29 22:44:28.5619632 +0300 +03 m=+0.054953001

1 messages sent successfully in 0 ms with 0 error(s) at 2025-06-29 22:44:29.5634994 +0300 +03 m=+1.056489201
2 messages sent successfully since 2025-06-29 22:44:28.5619632 +0300 +03 m=+0.054953001
...

After sending a limited number of messages—say 5 of them—press Ctrl+C to exit the process on that terminal.

Now, if you switch back to the other terminal, you will see that Director was receiving our Syslog messages, processing them through our pipeline, and sending them to our JSON file. As it was executing, it printed status messages on the screen like these:

[2025-06-29 22:44:31] [Information] [vmetric-director] [syslog-55555] Processing Syslog messages.. Number of Messages: 1
[2025-06-29 22:44:31] [Information] [vmetric-director] [syslog-55555] Completed processing of vmdb.syslog.55555.1751226271537836500.1.vmfl logs. Number of processed logs: 1
[2025-06-29 22:44:32] [Information] [vmetric-director] Completed processing of to_json target logs. Number of processed logs: 1
[2025-06-29 22:44:32] [Information] [vmetric-director] Completed processing of syslog logs on extract_user_event pipeline for syslog_to_json route. Number of processed logs: 2
...

Now go to the directory we have specified in location above, and check to see whether the file syslog-event-data.json has been created. If you ran the test without any errors, it should be there, and its contents should look like this:

{"Event":"John Doe says 'Hello world'"}
{"Event":"John Doe says 'Hello world'"}
{"Event":"John Doe says 'Hello world'"}
{"Event":"John Doe says 'Hello world'"}
{"Event":"John Doe says 'Hello world'"}

Monitoring

Check that the complete data stream worked by verifying:

• Output file syslog-event-data.json was created
• File contains JSON entries with transformed syslog data
• Logs in the logs directory show no errors

If the output file exists and contains the expected JSON format, your end-to-end data stream is working correctly.