Skip to main content

Library

Synopsis​

The Library is the dashboard area where you create and manage the reusable assets that pipeline processors reference: lookup tables, schemas, and grok patterns. Defining these assets once in the Library lets multiple pipelines share them instead of embedding the same data or definitions in every configuration.

Each asset type maps to a processor that consumes it:

  • Lookup tables provide the CSV data used by the Lookup and Enrich processors.
  • Schemas define the structures validated by the Check Schema processor.
  • Grok patterns supply the extraction patterns used by the Grok processor.

This page covers managing these assets through the web dashboard. For how each asset is referenced inside a pipeline, see the linked processor pages.

Director version requirement

The Library requires Director version 2.0 or later. If any director or director cluster in your fleet runs an earlier version, an alert at the top of the Library page lists the affected instances. Upgrade those directors to use Library assets in their pipelines.

Accessing the Library​

To open the Library:

  1. Click the hamburger menu in the top left corner.
  2. Select Content Management > Library.

The page opens with three tabs: Lookup Tables (default), Schemas, and Grok Patterns. Each tab lists the assets of that type and provides its own create flow and detail pages.

Working with Library Assets​

The three asset types share a common interaction model:

  • Assigned resources - Every overview table includes an Assigned resources column showing how many resources currently use the asset. Click the count to list the referencing pipelines (for schemas, referencing targets are listed as well where applicable).
  • Drawer - Click an asset's name in the table to open a detail drawer on the right side, summarizing the asset and offering quick actions.
  • Clone - Cloning opens the create flow pre-filled with the source asset's values, so you can save it under a new name.
  • Delete - An asset that is assigned to one or more resources cannot be deleted. A blocking modal lists the assigned resources; remove the asset from them first, then delete it.
  • Activity Logs - Each asset's detail page includes an Activity Logs tab with a searchable audit trail of actions performed on that asset.

The sections below cover the specifics of each asset type.

Lookup Tables​

A lookup table is a CSV dataset that pipelines match against to enrich or transform events.

Lookup Tables Table​

The table lists each lookup table with the following columns:

  • File Name - The lookup table's name. Click to open the detail drawer.
  • Type - The storage method, Memory or Disk.
  • Size - The size of the stored data.
  • Assigned resources - Count of resources referencing this table.

Above the table, the Search field filters by name, the Type dropdown filters by storage method (All, Memory, or Disk), and the Add new lookup table button opens the create flow. Pagination controls appear below the table.

The row action menu (the vertical ellipsis on each row) offers Manage lookup table, Clone lookup table, Download lookup table (exports the data as a CSV file), and Delete lookup table. The detail drawer offers Manage lookup table and Clone lookup table as quick actions.

Create a Lookup Table​

Click Add new lookup table and complete three steps:

Step 1 — Lookup Table Details

  • Lookup table name - Required. Between 3 and 64 characters.
  • Description - Optional.
  • Choose storage method - Required. Select Store in memory (recommended for smaller datasets that need fast access during execution) or Store on disk (recommended for larger datasets that do not need to be fully loaded into memory).

Step 2 — Lookup Table Configuration

Choose how to populate the table:

  • Upload CSV file - Upload a CSV file. Only the CSV format is accepted, with a maximum file size of 10 MB. A different format produces an error and blocks the next step.
  • Enter manually - Enter the data directly in a Monaco editor in CSV format. The first row holds the column headers, and values are comma-separated. The content is validated as CSV.

Step 3 — Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Lookup Table Details​

The detail page has three tabs:

  • General Overview - Displays the Name, Description, Type, Size, and the created and last-updated timestamps. Editable fields are saved from this tab.
  • Lookup Table Configuration - Shows the data and lets you edit it in two interchangeable views. Table view supports adding and deleting rows; Text view shows the data as CSV in a Monaco editor. Switching views warns that unsaved changes in the current view will be lost. The Replace file action overwrites the existing content with a newly uploaded CSV file.
  • Activity Logs - A searchable record of actions performed on the table.

The page header also provides an Actions menu containing Clone lookup table and Delete lookup table.

Schemas​

A schema defines a data structure that the Check Schema processor validates events against.

Schemas Table​

The table lists each schema with the following columns:

  • Schema Name - The schema's name. Click to open the detail drawer.
  • Description - The schema's description.
  • Version - An optional version label.
  • Assigned resources - Count of resources referencing this schema.

Above the table, the Search field filters by name and the Add new schema button opens the create flow. Unlike Lookup Tables, the Schemas tab has no Type filter. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage schema, Clone schema, and Delete schema.

Create a Schema​

Click Add new schema and complete three steps:

Step 1 — Schema Details

  • Schema name - Required. Between 3 and 64 characters.
  • Version - Optional. A label to track changes for your own reference.
  • Description - Optional.

Step 2 — Schema Configuration

  • Schema Format - Required. Select JSON, Parquet, or Avro. The format cannot be changed after creation.
  • Enter the schema definition in a Monaco editor. The definition is validated against the selected format.

Step 3 — Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Schema Details​

The detail page has three tabs:

  • General Overview - Displays and lets you edit the Name, Description, and Version, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
  • Schema Configuration - Shows the Schema Format (read-only) and the Schema Definition (editable).
  • Activity Logs - A searchable record of actions performed on the schema.

Grok Patterns​

A grok pattern is a reusable extraction pattern that the Grok processor applies to pull structured fields out of unstructured text.

Grok Patterns Table​

The table lists each grok pattern with the following columns:

  • File Name - The pattern's name. Click to open the detail drawer.
  • Description - The pattern's description.
  • Assigned resources - Count of resources referencing this pattern.

Above the table, the Search field filters by name and the Add new grok pattern button opens the create flow. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage pattern, Clone pattern, and Delete pattern.

Create a Grok Pattern​

Click Add new grok pattern and complete three steps:

Step 1 — Grok Pattern Details

  • File name - Required. Between 3 and 64 characters.
  • Description - Optional.

Step 2 — Grok Pattern Configuration

Enter the pattern content in a Monaco editor, for example:

%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status}

Step 3 — Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Grok Pattern Details​

The detail page has three tabs:

  • General Overview - Displays and lets you edit the Name and Description, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
  • Grok Pattern Configuration - Shows and lets you edit the Grok Pattern Content.
  • Activity Logs - A searchable record of actions performed on the pattern.