MQTT

MQTT Message Queue

Synopsis

Creates a target that publishes log messages to MQTT topics with support for batching, QoS levels, TLS encryption, and automatic retry logic.

Schema

- name: <string>
  description: <string>
  type: mqtt
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    topic: <string>
    client_id: <string>
    username: <string>
    password: <string>
    qos: <integer>
    retained: <boolean>
    clean_session: <boolean>
    auto_reconnect: <boolean>
    keep_alive: <integer>
    timeout: <integer>
    batch_size: <integer>
    max_retries: <integer>
    retry_delay: <integer>
    field_format: <string>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be mqtt
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	MQTT broker URL (e.g., tcp://localhost:1883, ssl://broker:8883, ws://broker:9001)
topic	Y	-	MQTT topic for message publishing
client_id	N	Auto-generated	MQTT client identifier (auto-generated if not specified)
username	N	-	MQTT username for authentication
password	N	-	MQTT password for authentication
timeout	N	30	Connection, write, and publish acknowledgement timeout in seconds
keep_alive	N	60	Keep-alive interval in seconds

MQTT Settings

Field	Required	Default	Description
qos	N	1	Quality of Service level: 1 (at least once) or 2 (exactly once). A 0 is read as unset, so the target publishes at QoS 1
retained	N	false	Retain messages on broker for new subscribers
clean_session	N	true	Start with clean session (discard previous session state)
auto_reconnect	N	true	Enable automatic reconnection on connection loss

Batch Configuration

Field	Required	Default	Description
batch_size	N	1000	Number of messages to batch before publishing (minimum 1)
max_retries	N	3	Maximum retry attempts for failed publish operations. Must be 0 or greater
retry_delay	N	1	Delay between retry attempts in seconds

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

TLS Configuration

Field	Required	Default	Description
tls.status	N	false	Enable TLS encryption

TLS material is resolved through the shared client builder, so these keys mean the same thing on every target that uses it. They are nested under a tls: block.

Field	Required	Default	Description
tls.verify	N	true	Verify the server certificate. On by default — set it to false only to accept an untrusted certificate, and only where you control the network path
tls.server_name	N	-	SNI hostname override for the handshake. Use it when the certificate's name does not match the address you connect to
tls.ca_name	N	-	CA bundle used to verify the server certificate. When unset the host trust store is used; when set it replaces the host trust store rather than adding to it
tls.cert_name	N*	-	Client certificate, for mutual TLS
tls.key_name	N*	-	Client private key, for mutual TLS
tls.passphrase	N	-	Passphrase for an encrypted client private key
tls.min_tls_version	N	tls1.2	Lowest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_version	N	tls1.3	Highest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)

* cert_name and key_name are individually optional but must be supplied together — a certificate without its key, or a key without its certificate, is a configuration error.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The MQTT target publishes log messages to MQTT topics using the Eclipse Paho MQTT client library. Messages are accumulated in batches and published when the batch size is reached. Each message is published with automatic retry logic and configurable Quality of Service levels.

The target maintains a persistent connection to the MQTT broker with automatic reconnection capabilities. Connection options include configurable timeouts, keep-alive intervals, and session management.

Quality of Service Levels

• QoS 0 (At most once): Fire-and-forget, no acknowledgment, lowest overhead
• QoS 1 (At least once): Acknowledged delivery, message may be duplicated
• QoS 2 (Exactly once): Guaranteed single delivery, highest overhead

note

Setting qos: 0 does not select QoS 0. A zero in a numeric field is read as unset, so the target falls back to the default of QoS 1.

Prerequisites

1. A running MQTT broker (e.g., Mosquitto, HiveMQ, AWS IoT Core)
2. Network connectivity to the MQTT broker
3. Valid authentication credentials if the broker requires authentication
4. TLS certificates if using encrypted connections

note

The client_id is auto-generated if not specified. For persistent sessions (clean_session: false), use a consistent client ID across reconnections.

warning

Both tls.cert_name and tls.key_name must be provided together when using client certificate authentication. Providing only one will result in a configuration error.

note

The retained flag causes the broker to store the last message on the topic, delivering it to new subscribers immediately. Use with caution for high-frequency log data.

Examples

Basic

Minimum configuration for publishing to MQTT:

targets:
  - name: basic_mqtt
    type: mqtt
    properties:
      url: "tcp://localhost:1883"
      topic: "logs/system"

With Authentication

Configuration with username/password authentication:

targets:
  - name: authenticated_mqtt
    type: mqtt
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/application"
      username: "logger"
      password: "secure_password"

With TLS

Configuration with TLS encryption:

targets:
  - name: secure_mqtt
    type: mqtt
    properties:
      url: "ssl://mqtt.example.com:8883"
      topic: "logs/secure"
      username: "logger"
      password: "secure_password"
      tls:
        status: true
        verify: true
        min_tls_version: "tls1.2"

With Client Certificate

Configuration using mutual TLS with client certificates:

targets:
  - name: mtls_mqtt
    type: mqtt
    properties:
      url: "ssl://mqtt.example.com:8883"
      topic: "logs/secure"
      tls:
        status: true
        verify: true
        cert_name: "client.pem"
        key_name: "client-key.pem"
        min_tls_version: "tls1.2"

High Reliability

Configuration with QoS 2 for guaranteed delivery:

targets:
  - name: reliable_mqtt
    type: mqtt
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/critical"
      qos: 2
      clean_session: false
      client_id: "datastream-logger-001"

Retained Messages

Configuration with retained messages for topic state:

targets:
  - name: retained_mqtt
    type: mqtt
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/status"
      retained: true
      qos: 1

Custom Batch Size

Configuration with custom batching and retry settings:

targets:
  - name: batch_mqtt
    type: mqtt
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/highvolume"
      batch_size: 5000
      max_retries: 5
      retry_delay: 2
      timeout: 60

With Normalization

Configuration using field normalization:

targets:
  - name: normalized_mqtt
    type: mqtt
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/ecs"
      field_format: "ecs"

WebSocket Connection

Configuration using WebSocket transport:

targets:
  - name: ws_mqtt
    type: mqtt
    properties:
      url: "ws://mqtt.example.com:9001"
      topic: "logs/websocket"
      username: "logger"
      password: "secure_password"

With Pipeline

Using a pipeline for additional log processing:

targets:
  - name: pipeline_mqtt
    type: mqtt
    pipelines:
      - enrich_logs
    properties:
      url: "tcp://mqtt.example.com:1883"
      topic: "logs/enriched"

Troubleshooting

This section covers the errors you are most likely to see with the mqtt target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:. The status only reflects the connection to the broker, so a message the broker accepts and then discards leaves it green.

See Target Delivery Errors for how Director logs and retries target failures.

Which permission or setting is missing?

Match the error against this table first. Which right or setting is missing depends on how your broker authenticates clients.

Error text	What is missing	Where to fix it
failed to connect to MQTT broker: bad user name or password	A username and password the broker accepts, or a broker that accepts anonymous clients	username and password on the target, and the broker's own user database
failed to connect to MQTT broker: not Authorized	Permission to connect for this user, client certificate or client identifier	The broker's access rules. On a managed IoT broker this is typically a policy that allows a connect action for this client
failed to connect to MQTT broker: identifier rejected	A client identifier the broker accepts	Set a short explicit client_id. Brokers that only speak the older protocol level typically reject identifiers longer than 23 characters, and the generated one is longer
Nothing in the log, and nothing arrives on the topic	A rule that allows this client to publish to topic	The broker's access rules. On a managed IoT broker this is typically a policy that allows a publish action on that topic
failed to connect to MQTT broker: network Error : x509: certificate signed by unknown authority	Trust for the certificate the broker presents	tls.ca_name, pointing at the CA that signed the broker certificate
failed to connect to MQTT broker: network Error : EOF	A TLS scheme in url. Setting tls.status: true does not switch the transport by itself	Change url from tcp:// to ssl:// and use the broker's TLS port

Retained publishing can need a right of its own. When retained: true, a broker that restricts retained messages typically refuses the publish even though ordinary publishing to the same topic works.

"failed to connect to MQTT broker: bad user name or password"

[Error] [director] [target-<target id>] [authenticated_mqtt] Failed to reinitialize target "authenticated_mqtt" (attempt 6). Reason: failed to connect to MQTT broker: bad user name or password

Cause: the broker refused the credentials. Three triggers are common. username or password is wrong, the broker does not accept anonymous clients and neither field is set, or password holds a reference to an environment variable that resolved to an empty value.

Fix: correct username and password, and confirm the same pair works from another MQTT client against the same host and port. If the value comes from an environment variable, check that it is set for the account the Director service runs under.

The target never connects, so nothing is published. Incoming data waits in the Director queue and is delivered once the credentials are accepted.

"failed to connect to MQTT broker: not Authorized"

Failed to reinitialize target "authenticated_mqtt" (attempt 2). Reason: failed to connect to MQTT broker: not Authorized

Cause: the broker accepted the credentials but does not allow this identity to connect. The rule that denies it typically matches on the user, on the client certificate, or on the client identifier.

Fix: grant connect permission on the broker for that user, and for the client identifier the target presents. When the broker rules match on the client identifier, set client_id explicitly. The generated identifier changes every time the target starts, so a rule written against it stops matching after the next restart.

Nothing is published while this persists, and queued data is delivered once the rule is in place.

"connection refused", "no such host" and "connection to MQTT broker timed out"

Failed to reinitialize target "basic_mqtt" (attempt 1). Reason: failed to connect to MQTT broker: network Error : dial tcp mqtt.example.com:1883: connect: connection refused
Failed to reinitialize target "basic_mqtt" (attempt 2). Reason: failed to connect to MQTT broker: network Error : dial tcp: lookup mqtt.example.com: no such host
Failed to reinitialize target "basic_mqtt" (attempt 4). Reason: connection to MQTT broker timed out

Cause: the broker was not reached at all. connection refused means nothing is listening on that port. no such host means the host name did not resolve. connection to MQTT broker timed out means no answer arrived within timeout seconds, which is what a firewall that drops the packets, or a broker that accepts the socket and never completes the handshake, produces.

Three more shapes come from the url value itself:

• network Error : dial tcp: address mqtt.example.com: missing port in address means the port is missing. No default port is added, so url must always carry a host and a port.
• failed to connect to MQTT broker: no servers defined to connect to means the URL could not be parsed, usually because of a space in the host or an unbalanced bracket, and was discarded before the connection was attempted.
• network Error : unknown protocol means the scheme is not one of tcp, ssl, ws and wss.

Fix: check reachability from the Director host to the broker host and port, correct the URL, and allow outbound TCP on the port you use. The usual ports are 1883 for plain connections, 8883 for TLS, and 9001 or 443 for WebSocket. Raise timeout only when the broker is known to be slow to answer.

Nothing is published while the target cannot connect, and queued data is delivered after it does.

"x509: certificate signed by unknown authority" and other certificate errors

Failed to reinitialize target "secure_mqtt" (attempt 3). Reason: failed to connect to MQTT broker: network Error : x509: certificate signed by unknown authority
Failed to reinitialize target "secure_mqtt" (attempt 1). Reason: failed to connect to MQTT broker: network Error : x509: certificate is valid for mqtt.example.com, not 10.0.0.12
Failed to reinitialize target "secure_mqtt" (attempt 2). Reason: failed to connect to MQTT broker: network Error : remote error: tls: bad certificate

Cause: the TLS handshake failed, and the tail of the line names the check that failed.

• certificate signed by unknown authority: the broker certificate was issued by a CA the Director host does not trust.
• certificate is valid for ..., not ...: url names a host the certificate does not cover, which is common when you connect by IP address.
• remote error: tls: bad certificate or remote error: tls: certificate required: the broker wants a client certificate and none was sent.
• remote error: tls: protocol version not supported: the configured version range excludes the versions the broker offers.

Fix: set tls.ca_name to the issuing CA in PEM form, connect by the name in the certificate or set tls.server_name to it, add tls.cert_name and tls.key_name for mutual TLS, and widen tls.min_tls_version and tls.max_tls_version. Setting tls.verify: false skips the check and leaves the connection unverified, so use it only to confirm a diagnosis.

A reason that starts with failed to load TLS certificate: is a different problem. The handshake never happened, because the material could not be read at startup. The two most common are ca_name: "certs/broker-ca.pem" could not be resolved (env/vault token, inline PEM, or a path under the service root), which means the path is wrong or lies outside the service root, and ca_name "certs/broker-ca.pem" contains no valid PEM certificate(s), which means the file is not PEM. An encrypted key with a missing or wrong tls.passphrase reports decrypt PKCS#8 private key (wrong passphrase?).

Queued data waits through all of these, and none of it is dropped.

"network Error : EOF" right after connecting

Failed to reinitialize target "secure_mqtt" (attempt 1). Reason: failed to connect to MQTT broker: network Error : EOF

Cause: the socket opened and the broker closed it without answering. The usual reason is the scheme. The scheme in url decides whether TLS is used, and tls.status does not. A tcp:// URL sends plaintext to a TLS port, and the broker hangs up. The same line appears when the port runs something other than MQTT, and when the broker drops the connection instead of answering it.

The mirror image of that mistake reports network Error : tls: first record does not look like a TLS handshake, which is an ssl:// URL pointed at a plain port. On WebSocket, a wrong path or a proxy in front of the broker reports network Error : websocket: bad handshake.

Fix: match the scheme to the listener. Use ssl://mqtt.example.com:8883 for a TLS listener, tcp://mqtt.example.com:1883 for a plain one, and ws:// or wss:// for a WebSocket listener, whose path is often /mqtt. Keep tls.status: true so that the TLS options are built, and note that ssl:// with tls.status: false still uses TLS, with the system trust store and verification on, ignoring every tls.* field.

Nothing is published until the connection succeeds, and queued data follows it.

"published 0/1000 messages, 1000 failed, last error: publish timeout"

[Error] [director] [target-<target id>] [batch_mqtt] Sender worker 3 execute() failed for payloads/batch_mqtt/000418.vmf: target broken: failed to finalize target cache: published 0/1000 messages, 1000 failed, last error: publish timeout
Sender worker 0 Finalize failed on flush for target "batch_mqtt": published 412/1000 messages, 588 failed, last error: not Connected

Cause: the connection was up when the batch started, and the broker did not acknowledge the messages. Each message waits up to timeout seconds for its acknowledgement. Common triggers are a broker that restarted while the client is still reconnecting, another client that connected with the same client_id and took over the session, a broker at its in-flight message limit, and a half-open network path. The two counts tell you how far the batch got before it stopped.

Three related texts appear in the same position. not Connected and MQTT client not connected mean the connection was already down, the second one before any message was attempted. publish was broken by timeout means the packet could not even be handed to the network within timeout seconds, which points at a broker that has stopped reading.

Fix: restore the broker, give every Director its own client_id, and leave auto_reconnect at true. Messages are published one at a time, so a single stalled batch can hold a worker for batch_size × (max_retries + 1) × timeout seconds. Lower batch_size or timeout when a stalled broker must not block a worker for that long.

The batch is redelivered in full. At QoS 1 or 2 the messages published before the failure are published again, so subscribers can see duplicates.

Large records and the broker's message size limit

Sender worker 1 execute() failed for payloads/batch_mqtt/000419.vmf: target broken: failed to finalize target cache: published 0/1000 messages, 1000 failed, last error: write tcp 10.0.0.5:51234->10.0.0.12:1883: write: connection reset by peer

Cause: each record is published as one MQTT message, and Director applies no size limit of its own. The protocol allows up to 268,435,455 bytes in a single packet, but brokers enforce far smaller limits, typically a configurable maximum message size, and typically 128 KB on managed IoT brokers. A broker typically reacts to an oversized message by closing the connection rather than by refusing the message, so the failure surfaces as a socket write error. The same texts follow a broker restart or a load balancer that resets idle connections.

Fix: compare your largest records against the broker's size setting, and read the broker log for the disconnect reason, which is the only place it is recorded. Reduce the record size in the pipeline, or raise the limit on the broker.

The broker never returns a refusal that Director can treat as permanent, so an oversized record is retried until either the record or the limit changes.

The broker accepts everything but nothing arrives on the topic

The target is connected, the counters climb, and no subscriber sees the data. Check these in order.

1. The broker allows the connection but not the publish. MQTT 3.1.1 has no negative acknowledgement for a refused publish. A broker whose rules deny publishing to topic typically acknowledges the message and discards it, so Director counts it as delivered and logs nothing. Check the publish rules for that exact topic on the broker, and confirm with a subscriber on the same topic.
2. debug.dont_send_logs is enabled. Events are processed by the pipeline and never published. When debug.status is also enabled, startup logs Log sending is disabled for this target (basic_mqtt). Logs will be processed by the pipeline but will not be sent to the target. Nothing else marks the difference.
3. max_retries is negative. The batch is acknowledged without a single publish attempt. See the entry below.
4. The connection drops and comes back on its own. With auto_reconnect: true the client reconnects quietly, and the reason for the drop, such as a missed keep-alive, appears only in the broker log. Batches fail with publish timeout while the session is away.

tip

Subscribe to the target's topic with any MQTT client, using credentials that are known to have read access. That separates a publishing problem from a subscriber problem in one step.

Zero and negative values for the numeric settings

Cause: a 0 in qos, timeout, keep_alive, batch_size or retry_delay is read as "not set", and the default is used instead. qos: 0 therefore publishes at QoS 1, and the validation message qos must be 0, 1, or 2 names a value you cannot actually select. Negative values behave in two different ways, and one of them loses data.

Fix: keep each of these fields at a value from the middle column.

Field	Safe values	What a zero or negative value does
qos	1 or 2	0 publishes at QoS 1. A negative value is refused at validation with qos must be 0, 1, or 2
max_retries	0 or greater	A negative value skips publishing entirely and reports the batch as delivered
batch_size	1 or greater	0 uses the default of 1000. A negative value fails startup with batch_size must be greater than 0, and the target retries that failure until you correct it
timeout, keep_alive, retry_delay	1 or greater	0 uses the default, which is 30, 60 and 1 respectively

warning

Never set max_retries to a negative value. No message is published, the batch is reported as delivered, and no error is logged. That data cannot be recovered. Remove the setting to use the default of 3.