OpenTelemetry Protocol (OTLP)

Network

Synopsis

Creates a target that forwards telemetry data to any OpenTelemetry Protocol (OTLP)-compatible receiver. Supports HTTP and gRPC transports, protobuf and JSON encoding, gzip compression, custom headers, TLS, and configurable retry logic.

note

This target has no GUI creation wizard. Configure it directly in the target YAML file.

Schema

- name: <string>
  description: <string>
  type: otlp
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    address: <string>
    port: <numeric>
    protocol: <string>
    path: <string>
    format: <string>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    grpc_passthrough: <boolean>
    headers:
      <key>: <value>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      server_name: <string>
      min_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be otlp
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y*	-	Full endpoint URL for HTTP (e.g. http://collector:4318) or host:port for gRPC
address	Y*	-	Remote host; alternative to url
port	Y*	-	Remote port; alternative to url

* = Provide url or both address and port. Exactly one form is required.

Transport

Field	Required	Default	Description
protocol	N	http	Wire transport: http or grpc
path	N	/v1/logs	HTTP URL path for the logs signal; leading / is added automatically. Metrics and traces are routed to /v1/metrics and /v1/traces automatically, regardless of this value. HTTP only.
format	N	protobuf	Encoding: protobuf or json. json is invalid with protocol: grpc.
compression	N	true	Enable gzip compression

Performance

Field	Required	Default	Description
timeout	N	30	Request or RPC timeout in seconds
batch_size	N	1000	Maximum events per batch; must be >= 1
max_retries	N	3	Maximum delivery attempts on failure; must be >= 0
retry_delay	N	1	Seconds between retry attempts

gRPC

Field	Required	Default	Description
grpc_passthrough	N	true	When true, sends raw OTLP bytes using content-subtype: raw-otlp (VirtualMetric-to-VirtualMetric only). When false, re-encodes as standard typed OTLP/gRPC for interoperability with any spec-compliant receiver.

Headers

Field	Required	Default	Description
headers	N	-	Map of string key-value pairs added to every HTTP request or gRPC metadata (e.g. authorization tokens, tenant IDs)

TLS

Field	Required	Default	Description
tls.status	N	false	Enable TLS

TLS material is resolved through the shared client builder, so these keys mean the same thing on every target that uses it. They are nested under a tls: block.

Field	Required	Default	Description
tls.verify	N	true	Verify the server certificate. On by default — set it to false only to accept an untrusted certificate, and only where you control the network path
tls.server_name	N	-	SNI hostname override for the handshake. Use it when the certificate's name does not match the address you connect to
tls.ca_name	N	-	CA bundle used to verify the server certificate. When unset the host trust store is used; when set it replaces the host trust store rather than adding to it
tls.cert_name	N*	-	Client certificate, for mutual TLS
tls.key_name	N*	-	Client private key, for mutual TLS
tls.passphrase	N	-	Passphrase for an encrypted client private key
tls.min_tls_version	N	tls1.2	Lowest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_version	N	tls1.3	Highest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)

* cert_name and key_name are individually optional but must be supplied together — a certificate without its key, or a key without its certificate, is a configuration error.

note

tls.max_tls_version is accepted but not applied on this target — the sender deliberately leaves the maximum version unset, so only tls.min_tls_version takes effect here.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The OTLP target delivers telemetry to any OpenTelemetry-compatible receiver — otel-collector, rotel, Grafana Alloy, vendor ingest endpoints, or another VirtualMetric Director.

Record translation

Records originating from this Director's own OTEL listener carry a JSON envelope with full Resource and Scope context. The target parses the envelope and reassembles each record as a one-record ExportLogsServiceRequest, preserving the original Resource and Scope end-to-end. This path is lossless.

Records from non-OTEL sources (TCP listener, syslog, etc.) are wrapped as a synthetic LogRecord with the body set to the raw message bytes and minimal resource attributes (service.name = the device name). This path is lossy by definition — non-OTEL sources have no OTLP context — but produces valid OTLP that any downstream collector accepts.

HTTP transport

HTTP is the default transport. Each worker maintains its own http.Client to avoid idle-connection contention. Batches are POSTed to endpoint + path. The Content-Type header is set to application/x-protobuf for format: protobuf and application/json for format: json. When compression: true, the body is gzip-compressed and Content-Encoding: gzip is set.

The signal is decided by the record, not by path. Metrics and traces from this Director's OTEL listener always go to /v1/metrics and /v1/traces on the same endpoint, whatever path is set to. The path field only overrides the route used for the logs signal, which is where a receiver with a non-standard logs route is accommodated. Records from non-OTEL sources are always sent as logs, so setting path to /v1/metrics or /v1/traces posts a logs body to a route that expects something else, and the receiver rejects it.

gRPC transport

When protocol: grpc, each worker maintains its own connection. The gRPC method is chosen per signal, so logs, metrics and traces are all carried over this transport: /opentelemetry.proto.collector.logs.v1.LogsService/Export, /opentelemetry.proto.collector.metrics.v1.MetricsService/Export, and /opentelemetry.proto.collector.traces.v1.TracesService/Export. The receiver must serve the service for every signal you send it.

grpc_passthrough: true (default): sends raw OTLP bytes using grpcraw.Codec with content-subtype: raw-otlp. This is the fast path from one Director to another, with no re-serialization. A standard OTLP receiver typically refuses this subtype, so set it to false for anything else.

grpc_passthrough: false: re-encodes each batch into the typed proto message (ExportLogsServiceRequest, etc.) using the standard gRPC codec and marshals with UnmarshalVT. This produces spec-compliant OTLP/gRPC that any receiver accepts, at the cost of one additional unmarshal+marshal per outbound batch. Use false for cross-vendor deployments.

Configuration validation

The following rules are enforced at config load:

• url or (address + port) must be provided; providing neither is rejected.
• format must be protobuf or json.
• format: json combined with protocol: grpc is rejected — OTLP/gRPC is protobuf-only on the wire.
• batch_size must be >= 1.
• max_retries must be >= 0.
• tls.cert_name and tls.key_name must both be present or both be absent.

Examples

Basic HTTP

Forwarding logs to an OTLP/HTTP collector with default settings...

targets: - name: otel_collector type: otlp properties: url: "http://otel-collector.example.com:4318"

JSON Format over HTTP

Sending telemetry as JSON-encoded OTLP to an HTTP endpoint...

targets: - name: otel_json type: otlp properties: url: "http://otel-collector.example.com:4318" format: json path: /v1/logs

gRPC Spec-Compliant

Delivering logs over gRPC to a third-party collector using typed OTLP encoding...

targets: - name: grpc_collector type: otlp properties: address: "otel-collector.example.com" port: 4317 protocol: grpc grpc_passthrough: false

TLS-Secured

Sending logs over HTTPS with mutual TLS client certificate authentication...

targets: - name: otel_secure type: otlp properties: url: "https://otel-collector.example.com:4318" tls: status: true verify: true cert_name: "client-cert.pem" key_name: "client-key.pem" min_tls_version: "tls1.2"

Performance Tuning

High-volume configuration with large batches, retry logic, and auth header...

targets: - name: otel_high_volume type: otlp properties: url: "https://ingest.example.com:4318" format: protobuf compression: true batch_size: 5000 timeout: 60 max_retries: 5 retry_delay: 2 headers: Authorization: "Bearer ${OTLP_TOKEN}" X-Tenant-ID: "tenant-prod"

Troubleshooting

This section covers the errors you are most likely to see with the otlp target, what causes each one, and how to fix it. Most of them come from the receiver rather than from Director, so the status code or the text after desc = is the receiver's own verdict on what you sent.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause. Lines that begin with [OTLP] also name the URL or the gRPC method that failed, which tells you exactly which endpoint to check.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

What the collector needs

This target has no identity model of its own. Everything it depends on lives on the receiver. Match a failure against this table before changing anything else.

The receiver must	When it applies	What you see when it is missing
Listen for the configured protocol on the port you set	Always	HTTP: connect: connection refused. gRPC: code = Unavailable with a dial error, or code = Unimplemented desc = unknown service ... when the port serves something other than OTLP
Serve the signal being sent	Always	HTTP status 404. gRPC code = Unimplemented desc = unknown service ...
Accept the credential in headers	When the receiver authenticates	HTTP status 401 or status 403. gRPC code = Unauthenticated or code = PermissionDenied
Accept gzip-compressed payloads	compression: true, which is the default	gRPC code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "gzip". Over HTTP a receiver typically answers status 415 or status 400
Accept application/json bodies	format: json	Over HTTP a receiver typically answers status 415 or status 400
Accept the size of one batch	Always	HTTP status 413. gRPC code = ResourceExhausted desc = grpc: received message larger than max (...)

There is no size limit on the Director side, so an oversized batch is only ever refused by the receiver. Lower batch_size when you see status 413 or ResourceExhausted.

"status 401" or "status 403" over HTTP

[Error] [director] [target-<target id>] [otel_high_volume] [OTLP] http raw send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_high_volume msglen=48213 err=status 401
[Error] [director] [target-<target id>] [otel_high_volume] Sender worker 3 execute() failed for <payload>: target broken: status 401

Cause: the receiver refused the request. 401 means it found no usable credential, 403 means it found one and would not accept it. The only credentials this target sends are the entries under headers, plus a client certificate when tls.cert_name and tls.key_name are set.

Fix: set the header the receiver expects, such as Authorization: "Bearer ${OTLP_TOKEN}" or a vendor API-key header. Two mistakes produce an empty header set with no error of their own, so the 401 is all you get:

• headers must be written as a YAML mapping. Written as a list of - key: value items it is read as empty, and no headers are sent at all.
• A ${...} or $secret{...} reference that resolves to nothing yields an empty header value.

Over gRPC the same problem arrives as rpc error: code = Unauthenticated desc = ... or code = PermissionDenied desc = ..., with wording the receiver chooses. Header names are lower-cased when they are sent as gRPC metadata, which matters for a receiver that compares them case-sensitively.

The payload is retried until the credential is accepted, and the queue grows in the meantime. Retries continue while Director runs and while the records are inside the queue retention limit, which is queue.limit seconds, 48 hours by default. With the In-memory storage tier they do not survive a Director restart.

"status 404" over HTTP, or the signal path appears twice

[Error] [director] [target-<target id>] [otel_collector] [OTLP] send-failed url=http://collector.example.com:4318/v1/logs/v1/logs file=<payload> target=otel_collector batch=1000 attempts=1 budget=4 lastStatus=404 lastErr=status 404

Cause: the request went to a route the receiver does not serve. Read the url= value in the log line before anything else. The target appends path to the endpoint you configured, so a url that already ends in /v1/logs produces /v1/logs/v1/logs, exactly as above. The other causes are a receiver that publishes OTLP on a different route, and a receiver that has the signal disabled.

Fix: set url to the base address only, for example http://collector.example.com:4318, and let path supply the route. When the receiver uses a non-standard route for logs, put that route in path and still keep url at the base. Retried until fixed, and nothing is dropped while you correct it.

"connection refused" or "no such host"

[Error] [director] [target-<target id>] [otel_collector] [OTLP] send-error url=http://collector.example.com:4318/v1/logs file=<payload> target=otel_collector batch=1000 attempt=0 err=Post "http://collector.example.com:4318/v1/logs": dial tcp <collector ip>:4318: connect: connection refused
[Error] [director] [target-<target id>] [otel_collector] Sender worker 5 execute() failed for <payload>: target broken: failed to finalize target cache: failed to send after 4 of 4 attempts: Post "...": connect: connection refused

Cause: nothing is listening where the target points, or the route to it is blocked. Starting the target opens no connection, so a wrong host or port never shows up as a startup failure, and the connection status stays green until the first delivery.

Fix: confirm the receiver is up and the port is reachable from the Director host. 4318 is the convention for OTLP/HTTP and 4317 for OTLP/gRPC, and swapping the two is the most common cause. For HTTP, url must carry a scheme: a value such as collector.example.com:4318 is accepted when the configuration loads and then fails on every send with unsupported protocol scheme. Check HTTP_PROXY, HTTPS_PROXY and NO_PROXY on the Director service if the traffic should not go through a proxy.

Over gRPC the same causes arrive as rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: ...". On that transport the endpoint is a host and port with no scheme, so an http:// prefix in url becomes part of the name being looked up, and the failure is typically reported as a name resolver error instead. Use address and port, or a url of the form collector.example.com:4317.

Retried until fixed. Data stays queued and is delivered once the receiver is reachable.

"x509: certificate signed by unknown authority" or "certificate is valid for ..."

[Error] [director] [target-<target id>] [otel_secure] [OTLP] send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_secure batch=1000 attempt=0 err=Post "https://collector.example.com:4318/v1/logs": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: Director does not trust the certificate the receiver presented.

• certificate signed by unknown authority: the issuing CA is not trusted. Note that tls.ca_name replaces the system trust store rather than adding to it, so a publicly issued certificate stops verifying the moment you point the target at a private CA.
• certificate is valid for ..., not ...: the name being connected to is not in the certificate. This is normal when you connect by IP address.
• certificate has expired or is not yet valid: renew it, or check the clock on the Director host.

Fix: set tls.ca_name to the CA that issued the receiver's certificate, and include the public CA bundle in it if the same Director also needs public endpoints. Set tls.server_name to a name the certificate carries when you connect by IP. Turning tls.verify off removes the check, and the protection with it.

A TLS setting that does not match the port looks different. Over HTTP, https:// against a plaintext port reports tls: first record does not look like a TLS handshake. Over gRPC, plaintext against a TLS port reports code = Unavailable with an error reading the server preface.

Certificate material is read when the target starts, so a bad file stops it before anything is sent:

[Error] [director] [target-<target id>] [otel_secure] Failed to reinitialize target "otel_secure" (attempt 4). Reason: failed to build TLS config: client certificate: decrypt PKCS#8 private key (wrong passphrase?): ...

Other reasons in that position name the material that could not be used, for example ca_name "..." contains no valid PEM certificate(s). PEM files must sit under the service root, or be supplied inline or through an environment or secret reference. One setting fails quietly: an unrecognized tls.min_tls_version falls back to tls1.2 with no warning, so use tls1.0 through tls1.3 exactly.

"Unimplemented ... unknown service ..." over gRPC

[Error] [director] [target-<target id>] [grpc_collector] [OTLP] grpc raw send-error method=/opentelemetry.proto.collector.logs.v1.LogsService/Export file=<payload> target=grpc_collector msglen=48213 err=rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.logs.v1.LogsService

Cause: the connection succeeded, but the server on that port does not offer the OTLP service named after method=. In practice the port is wrong. Pointing a protocol: grpc target at 4318, the OTLP/HTTP port, is the usual way to produce this. The other cause is a receiver that has the signal named in method= turned off.

Fix: set port to the receiver's OTLP/gRPC port, conventionally 4317, or enable that signal on the receiver. A close variant, desc = unknown method Export for service ..., means the service is there but the method is not, which points at a server that is not an OTLP receiver at all.

Retried until fixed, so the queue grows until the port is corrected. Nothing is dropped.

"Decompressor is not installed for grpc-encoding ..."

[Error] [director] [target-<target id>] [grpc_collector] [OTLP] grpc raw send-error method=/opentelemetry.proto.collector.logs.v1.LogsService/Export file=<payload> target=grpc_collector msglen=48213 err=rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "gzip"

Cause: compression defaults to true, so every call is gzipped and asks the receiver to decompress it. This receiver has no gzip decompressor enabled. The status code is Unimplemented, the same as the previous entry, so read the text after desc = to tell the two apart.

Fix: set compression: false on the target, or enable gzip on the receiver. Prefer enabling it on the receiver, since compression is the cheaper of the two on a busy link.

Over HTTP there is no dedicated error for the same situation. A receiver that will not take a gzipped body, or the application/json body produced by format: json, typically answers status 415 or status 400. When those appear and the endpoint is otherwise correct, try compression: false and format: protobuf in turn. Do not set Content-Type or Content-Encoding under headers to work around it. A custom value there replaces the one the target computed, and the receiver then sees a body that does not match its declared type.

Retried until fixed.

"context deadline exceeded" or "code = DeadlineExceeded"

[Error] [director] [target-<target id>] [otel_high_volume] [OTLP] send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_high_volume batch=5000 attempt=0 err=Post "https://collector.example.com:4318/v1/logs": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Cause: the request did not finish within timeout, which defaults to 30 seconds. Every attempt gets the full timeout, so one flush can take several times that before it gives up. A large batch, a slow or throttled receiver, and a firewall that drops packets silently all look the same here.

Fix: raise timeout, or lower batch_size so each request carries less. If every attempt times out at the same point and the receiver logs nothing at all, treat the route as blocked rather than slow and check the path from the Director host to the receiver. status 429 in the same place means the receiver is throttling instead, which Director retries on its own.

Setting timeout: 0 does not remove the limit. A zero is read as the default, and the same is true of batch_size, max_retries and retry_delay. Use max_retries: 1 for the fewest attempts.

The same rejection keeps coming back

Symptom: one payload fails with the same answer on every attempt, for example status 400, status 413 or status 422 over HTTP, or rpc error: code = InvalidArgument desc = ... over gRPC. The target reconnects between attempts, the delivered counter does not move, and the queue keeps growing. The connection status stays green, because the connection itself is fine.

Cause: no answer from a receiver is treated as final. A payload the receiver will never accept is redelivered rather than discarded, so it returns at a steady pace until you change something.

Fix: change whatever the receiver is objecting to.

• status 413, or ResourceExhausted over gRPC: lower batch_size. The gRPC ceiling on the receiving side is 4 MB by default.
• status 400 or status 422: check path and format against what the receiver accepts, then look at the records the pipeline produces. Sending a logs body to /v1/metrics is a common cause, because records from non-OTEL sources are always sent as logs.
• InvalidArgument from a third-party receiver: set grpc_passthrough: false. The default of true uses an encoding that only another Director reads.

warning

Nothing is lost, but nothing queued behind the refused payload moves either. Queued data is retained only for the configured queue limit, so correct the cause rather than waiting it out. See Persistent Storage.

The target is healthy but records are missing

Check these in order.

1. debug.dont_send_logs is enabled. Events are processed and then discarded before any request is made. With debug.status: true, startup logs Log sending is disabled for this target once, naming the target. With debug.status: false nothing is logged at all. Remove the flag.

2. The receiver accepted the call but refused some records. Look for [OTLP] partial_success in the log:

   [Warning] [director] [target-<target id>] [otel_collector] [OTLP] partial_success signal=logs target=otel_collector rejected=2 message="<receiver message>"

   The text after message= is the receiver's own explanation, usually an attribute or schema limit. Those records are counted as dropped and are never retried, because the call itself succeeded. Fix the records upstream, or raise the limit on the receiver.

3. The receiver's answer could not be read. Only the first 16 KB of a response is examined. A receiver that reports its rejections in a longer body, or in a shape this target does not recognize, leaves the whole batch counted as delivered. When the Director counters say everything arrived and the receiver disagrees, the receiver's own logs are the only place left to look.

4. A pipeline rewrote the OTLP envelope. Records from this Director's OTEL listener carry a JSON envelope. A processor that renames or removes its signal, resource, scope or logRecord keys makes the record unusable. It is skipped with a warning, and it is not counted as a drop:

   [Warning] [director] [target-<target id>] [otel_collector] Skipping malformed record: unsupported signal ""

   Leave those keys intact in any pipeline attached to this target.

5. The records went out as a different signal than you expect. Only records from the OTEL listener carry a signal of their own. Everything else, syslog and TCP sources included, is sent as logs whatever path says.