Skip to main content

Rapid7 InsightIDR

SIEM Platform

Synopsis​

The Rapid7 InsightIDR target forwards events to a Rapid7 InsightIDR log HTTP ingestion endpoint over HTTPS. It handles batching and delivery — the event payload is expected to already be a complete JSON document in InsightIDR's schema when it reaches the target.

Schema​

- name: <string>
  description: <string>
  type: rapid7
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    api_key: <string>
    batch_size: <integer>
    timeout: <integer>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be rapid7
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Connection​

FieldRequiredDefaultDescription
urlY-Rapid7 InsightIDR log HTTP ingestion endpoint. The ingestion token is embedded in the URL, e.g. `https://<region>.data.logs.insight.rapid7.com/v1/input/hook/<TOKEN>`
api_keyN-Optional value sent as the x-api-key header, for endpoints that require it in addition to the embedded token
max_bytesN5242880 (5 MB)Maximum size of a single request body or batch, in bytes
note

max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:

  • An explicit max_bytes: 0 IS honoured and removes the ceiling on targets that have no ceiling of their own. The value is read with Int64, which returns what you wrote — unlike max_size, which is read with MustInt64 and silently substitutes its default for a zero. Where a target has a fixed ceiling of its own, its page says so and a zero is clamped to that ceiling instead.
  • A negative value is rejected at configuration time with max_bytes must not be negative.

The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.

Batch Configuration​

FieldRequiredDefaultDescription
batch_sizeN1000Maximum events per POST, serialized as newline-delimited JSON
timeoutN30HTTP client timeout, in seconds

Processing​

FieldRequiredDefaultDescription
field_formatN-Optional target-side normalization format applied before delivery. Typically left empty, since the payload already arrives in InsightIDR's JSON schema from upstream processing. See applicable Normalization section

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging
debug.dont_send_logsNfalseProcess logs but don't send to target (testing)

Details​

This target is transport-only: it does not render InsightIDR's log schema itself. It expects each event to already be a complete JSON document when it reaches the target, buffers events in memory, and delivers them as one HTTP POST — newline-delimited JSON, one document per line — once batch_size is reached or during finalization.

Endpoint and Authentication​

The url is the InsightIDR log HTTP ingestion endpoint; the ingestion token is embedded directly in the URL path. api_key is optional and only needed if the endpoint additionally requires an x-api-key header. Configuration validation rejects a target with no url.

Batching and Retry​

Events buffer until batch_size is reached, then POST as a single request. A failed POST does not retry the batch internally: the buffer is cleared unconditionally after Finalize, and delivery failure is surfaced to the sender pool, which handles redelivery through the JetStream layer. Retaining the batch locally in addition to that redelivery would duplicate events at the destination. Error responses include up to the first 1 MB of the response body for troubleshooting.

note

The target does not render InsightIDR's JSON schema — it only batches and delivers the payload it receives. Format the event as InsightIDR-compatible JSON before it reaches this target.

Examples​

Basic Configuration​

Sending JSON events to a Rapid7 InsightIDR log HTTP endpoint...

targets:
  - name: rapid7
    type: rapid7
    properties:
      url: "https://us.data.logs.insight.rapid7.com/v1/input/hook/TOKEN"

With API Key Header​

Adding an x-api-key header for endpoints that require it alongside the embedded token...

targets:
  - name: rapid7-authenticated
    type: rapid7
    properties:
      url: "${RAPID7_LOG_ENDPOINT}"
      api_key: "${RAPID7_API_KEY}"

High-Volume Configuration​

Increasing batch size and timeout for high-volume ingestion...

targets:
  - name: rapid7-high-volume
    type: rapid7
    properties:
      url: "${RAPID7_LOG_ENDPOINT}"
      batch_size: 5000
      timeout: 45

Troubleshooting​

The rapid7 target behaves the same way as the datadog target and reports the same errors for the same causes: a refused credential, a wrong or unreachable endpoint, certificate failures, rejected bodies, oversized records, throttling, and the silent failures that leave the target looking healthy. Use the Troubleshooting section of the Datadog target for the full list of errors, causes and fixes, and Target Delivery Errors for how Director logs and retries target failures.

Read those entries with this target's type in place of datadog. A refusal from the ingestion endpoint is logged as rapid7 api returned status <code>: followed by the response body, up to its first 1 MB.

What differs​

  • The credential lives in url. The ingestion token is part of the path. api_key is optional and only adds an x-api-key header for endpoints that ask for one in addition to the token, so a refused request is usually a problem with the URL rather than with api_key.

  • A failed request exposes that token. A DNS, connection, proxy, TLS or timeout error puts the whole endpoint URL, ingestion token included, into the log line and into the target's connection status. See Log lines and connection status can contain the full endpoint URL on the Datadog page for how to treat those logs, and when to rotate the token.

  • max_bytes defaults to 5242880 (5 MB), the same as Datadog, but InsightIDR publishes no request-size limit of its own. InsightIDR still applies its own request and event limits, so keep a ceiling rather than removing it, and check the current limits before raising it. On the other two targets the same change only turns an oversized-record rejection into a 413.

  • The body is newline-delimited JSON, one document per line. An event whose own text contains a raw newline is split into two documents at the receiver, so make sure the pipeline emits single-line JSON.

  • A missing url is reported as rapid7 target requires url (the InsightIDR log HTTP endpoint), not as a missing key.

warning

A log line or connection status that shows this target's url carries a working ingestion token. Remove it before sharing the line, and if it has already been shared, create a new log ingestion endpoint, point url at it, and delete the old one.