Library

Standard and Enterprise

Synopsis

The Library is where you create and manage the reusable assets that pipeline processors reference: lookup tables, schemas, and grok patterns. Defining these assets once in the Library lets multiple pipelines share them instead of embedding the same data or definitions in every configuration.

Each asset type maps to a processor that consumes it:

• Lookup tables provide the CSV data used by the Lookup and Enrich processors.
• Schemas define the structures validated by the Check Schema processor.
• Grok patterns supply the extraction patterns used by the Grok processor.

The Library also holds samples — reusable log data that device simulation, the pipeline debugger and prediction draw on rather than asking you to paste the same records in again. Samples are covered separately under Library: Samples.

The Library also holds sensitive data rules. Each rule finds one kind of sensitive value, such as a credit card number or an email address. Masked captures in live data, console logs and Search hide the values that the enabled rules find. Sensitive data rules are covered under Library: Sensitive Data Rules.

This page covers managing these assets through the web interface. For how each asset is referenced inside a pipeline, see the linked processor pages.

Director version requirement

The Library requires Director version 2.0 or later. If any director or director cluster in your fleet runs an earlier version, an alert at the top of the Library page lists the affected instances. Upgrade those directors to use Library assets in their pipelines.

Sensitive data rules reach a director from version 2.5.1. A director on an older version doesn't receive them and keeps using the rules built into it. The Sensitive Data Rules tab has its own alert, Some directors use the built-in rules only, which names those directors and director clusters.

Accessing the Library

To open the Library:

1. Click the hamburger menu in the top left corner.
2. Select Content Management > Library.

The page opens with seven tabs: Lookup Tables (default), Schemas, Grok Patterns, Scripts, Samples, SNMP Vendors and Sensitive Data Rules. Each tab lists the assets of that type and provides its own create flow. Every tab except SNMP Vendors also has detail pages.

This page covers the first three, and SNMP Vendors in SNMP Vendors. Samples and Sensitive Data Rules each have a page of their own: see Library: Samples and Library: Sensitive Data Rules. Scripts is not covered here. To run a Library script from a pipeline, and for what signing it changes, see the Script processor.

Working with Library Assets

The three asset types share a common interaction model. SNMP Vendors does not: it has no detail page and no Assigned resources column, so of the following only Bulk actions applies to it. Its row actions, Delete included, are listed under SNMP Vendors Table.

Sensitive Data Rules has no Assigned resources column, and nothing blocks hiding or deleting a rule because something uses it. The drawer, cloning, Activity Logs and Bulk actions work as described below.

• Assigned resources - Every overview table includes an Assigned resources column showing how many resources currently use the asset. Click the count to list the referencing pipelines (for schemas, referencing targets are listed as well where applicable).
• Drawer - Click an asset's name in the table to open a detail drawer on the right side, summarizing the asset and offering quick actions.
• Clone - Cloning opens the create flow pre-filled with the source asset's values, so you can save it under a new name.
• Delete - An asset that is assigned to one or more resources cannot be deleted. A blocking modal lists the assigned resources; remove the asset from them first, then delete it.
• Activity Logs - Each asset's detail page includes an Activity Logs tab with a searchable audit trail of actions performed on that asset.
• Bulk actions - Every tab, Scripts, Samples, SNMP Vendors and Sensitive Data Rules included, has checkboxes for acting on several items at once. See Bulk Actions.

Names with spaces

A director finds a Library entry by its name with every space removed: a lookup table named Geo IP ships to the director as GeoIP.csv, and a pipeline reaches it as GeoIP. Reference a lookup table, grok pattern, schema or script by its name without spaces. A reference that keeps the spaces still links the entry, but the processor on the director looks for a name it does not have and fails, silently when ignore_failure is set. Names stay case-sensitive; only spaces are removed.

Two entries of the same type cannot have names that differ only by spaces, such as Geo IP and GeoIP, because they would ship to the same file on a director. Creating or renaming an entry onto such a name is refused as a name already in use. Pairs created before this check keep working as before: a reference that names one of them exactly links that one, and a reference that matches neither exactly, such as GeoIP for Geo IP and Ge oIP, links and ships neither, and the platform logs a warning. Rename one of them.

Lookup files given as a path ending in .csv, built-in schema models and inline schema definitions are not affected.

Bulk Actions

Every Library tab — Lookup Tables, Schemas, Grok Patterns, Scripts, Samples, SNMP Vendors and Sensitive Data Rules — has a checkbox on each row. Selecting rows replaces the toolbar with a bar that says how many are selected ("3 schemas selected"), offers the actions that fit the selection, and has Cancel to clear it.

A bulk action works the same way on every tab:

1. Select rows with their checkboxes, or with the header checkbox for the whole page. The selection is kept when you move to another page.
2. The bar shows only the actions that apply to something selected. An action that applies to only part of the selection shows how many it will act on: Sign (2) with three scripts selected, one already signed. At most 500 items can be acted on at once; above that, the bar says so.
3. Clicking an action opens a confirmation. It lists the items the action will act on, and under Left out (n) the items it will skip, with the reason.
4. Confirming runs the action on each item. The button shows progress ("Deleting 4 of 12…"), and the dialog cannot be closed until it finishes.
5. When everything succeeds, a notification confirms it. If anything failed, a result dialog lists what failed, grouped by the reason the server gave, and what was done (collapsed). Either way, the items the action succeeded on, and those it left out, are cleared from the selection. What failed stays selected, so it can be retried once the cause is fixed. Items the action did not apply to also stay selected, ready for the next action.

The items left out are the ones the row menu would refuse:

Reason (as shown)	When
In use by pipelines, devices or targets.	Delete or Hide on an item a resource references
Managed by a repository sync source.	The item comes from a GitOps repository
Already signed.	Sign on a signed script
Used by devices.	Delete on a sample devices use
Platform samples can't be deleted.	Delete on a platform sample

Each action needs the permission of the row action it repeats. A user who may run none of a tab's actions gets no checkboxes there. With Library read alone, that leaves only Generate template on Schemas.

Tab	Bulk actions	What each acts on (the rest is left out, with its reason)
Lookup Tables	Download (Library edit, as in the row menu), Delete	Download saves each table as a CSV file, all in one ZIP file (lookup-tables.zip). Delete leaves out tables in use.
Schemas	Generate template, Reset to platform, Hide, Delete, Restore	Generate template: visible schemas (see Several Schemas at Once). Reset to platform: edited platform schemas. Hide: platform schemas. Delete: your own schemas. Restore: hidden schemas. Repository-managed schemas, and for Hide and Delete schemas in use, are left out.
Grok Patterns	Reset to platform, Hide, Delete, Restore	As for schemas.
Scripts	Sign, Delete	Sign leaves out signed scripts. Delete leaves out scripts in use or managed by a repository.
Samples	Delete	Leaves out platform samples and samples devices use.
SNMP Vendors	Hide, Reset to built-in, Delete, Restore	The entries the row menu offers each action on.
Sensitive Data Rules	Reset to platform, Hide, Delete, Restore	Reset to platform: built-in rules you have changed. Hide: built-in rules. Delete: your own rules. Restore: hidden rules. No rule is left out for being in use.

A selection can mix kinds. For example, a platform schema and one of your own show both Hide (1) and Delete (1), and each acts only on its own rows.

The sections below cover the specifics of each asset type.

Lookup Tables

A lookup table is a CSV dataset that pipelines match against to enrich or transform events.

A lookup table can also be generated on a schedule from a recipe — a script, a SQL query, or an HTTP endpoint — instead of being uploaded or entered by hand. See Lookup Sources.

Lookup Tables Table

The table lists each lookup table with the following columns:

• File Name - The lookup table's name. Click to open the detail drawer.
• Type - The storage method, Memory or Disk.
• Size - The size of the stored data.
• Assigned resources - Count of resources referencing this table.

Above the table, the Search field filters by name, the Type dropdown filters by storage method (All, Memory, or Disk), and the Create new lookup table button opens the create flow. Pagination controls appear below the table.

The row action menu (the vertical ellipsis on each row) offers Manage lookup table, Clone lookup table, Download lookup table (exports the data as a CSV file), and Delete lookup table. The detail drawer offers Manage lookup table and Clone lookup table as quick actions.

Create a Lookup Table

Click Create new lookup table and complete three steps:

Lookup Table Details

• Lookup table name - Required. Between 3 and 64 characters.
• Description - Optional.
• Choose storage method - Required. Select Store in memory (recommended for smaller datasets that need fast access during execution) or Store on disk (recommended for larger datasets that do not need to be fully loaded into memory).

Lookup Table Configuration

Choose how to populate the table:

• Upload CSV file - Upload a CSV file. Only the CSV format is accepted, with a maximum file size of 10 MB. A different format produces an error and blocks the next step.
• Enter manually - Enter the data directly in a Monaco editor in CSV format. The first row holds the column headers, and values are comma-separated. The content is validated as CSV.

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Lookup Table Details

The detail page has three tabs:

• General Overview - Displays the Name, Description, Type, Size, and the created and last-updated timestamps. Editable fields are saved from this tab.
• Lookup Table Configuration - Shows the data and lets you edit it in two interchangeable views. Table view supports adding and deleting rows; Text view shows the data as CSV in a Monaco editor. Switching views warns that unsaved changes in the current view will be lost. The Replace file action overwrites the existing content with a newly uploaded CSV file.
• Activity Logs - A searchable record of actions performed on the table.

The page header also provides an Actions menu containing Clone lookup table and Delete lookup table.

Schemas

A schema defines a data structure that the Check Schema processor validates events against.

Schemas Table

The table lists each schema with the following columns:

• Schema Name - The schema's name. Click to open the detail drawer.
• Description - The schema's description.
• Version - An optional version label.
• Format - The schema's format: JSON, Parquet or Avro.
• Assigned resources - Count of resources referencing this schema.

The platform's own table schemas — the Microsoft Sentinel and Defender XDR tables, and the OCSF classes — are all Parquet. A schema you create can be any of the three.

Above the table, the Search field filters by name, the Origin filter narrows the list to All, Platform, Custom or Hidden schemas, and the Create new schema button opens the create flow. Unlike Lookup Tables, the Schemas tab has no Type filter. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage schema and Clone schema; the row action menu also offers Generate template (see Generate a Template). The destructive action depends on where the schema came from: a schema you created offers Delete schema, while a platform schema offers Hide schema instead, since the platform's own schemas cannot be deleted. A hidden schema offers Restore schema, and a platform schema you have edited also offers Reset to platform, which replaces your changes with the platform's current version under the same name.

Create a Schema

Click Create new schema and complete three steps:

Schema Details

• Schema name - Required. Between 3 and 64 characters.
• Version - Optional. A label to track changes for your own reference.
• Description - Optional.

Schema Configuration

• Schema Format - Required. Select JSON, Parquet, or Avro. The format cannot be changed after creation.
• Enter the schema definition in a Monaco editor. The definition is validated against the selected format. The JSON definition syntax is documented under Avro and Parquet.

A registered schema is referenced by name from the Enforce Schema and Check Schema processors.

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Schema Details

The detail page has three tabs:

• General Overview - Displays and lets you edit the Name, Description, and Version, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
• Schema Configuration - Shows the Schema Format (read-only) and the Schema Definition (editable).
• Activity Logs - A searchable record of actions performed on the schema.

Generate a Template

A schema can be turned into what Microsoft Sentinel needs to ingest its table through the Logs Ingestion API: the Data Collection Rule (DCR), or an ARM or Bicep template. Each creates the custom _CL table too, unless you say it exists already. The template is computed on demand and stored nowhere.

Generate template needs the Library read permission (LIBRARY_READ). It is not offered on schemas listed under the Hidden origin filter. JSON, Parquet and Avro schemas all convert.

1. In the row action menu, click Generate template. The Generate template drawer opens on the right.

2. Under Generate, pick Data Collection Rule (DCR) (the default), ARM template or Bicep template.

3. Under Microsoft Sentinel, fill in what you need. Every field is optional:

   Field	Default	Meaning
   Table name	The schema's name	A name ending in _CL is a custom table. Any other name must be an Azure table that accepts the Logs Ingestion API.
   Stream name	Custom- and the table name less _CL	The stream the rule declares, which a sender posts to.
   Data collection rule name	vmetric-, the table name less _CL in lower case, then -dcr (cut to 64 characters)	The rule's name, and the file's.
   Data collection endpoint	None: a Direct rule, which has an ingestion endpoint of its own	The endpoint's name (in the rule's resource group) or its resource ID.
   Workspace name	A <WorkspaceName> placeholder in a DCR; a template parameter in ARM and Bicep	The Log Analytics workspace (Microsoft Sentinel's).
   Subscription ID	A <SubscriptionId> placeholder in a DCR	Used in the DCR's resource IDs and in the deploy command's --subscription.
   Resource group	A <ResourceGroup> placeholder	The workspace's resource group, where the template deploys the rule and the table.
   Location	A <Location> placeholder in a DCR; the resource group's location in ARM and Bicep	The workspace's region, which the rule must be in.
   Create the custom tables (custom tables only)	On	The template creates each table whose name ends in _CL before the rule. Turn this off if these tables already exist in the workspace.
   Table plan (custom tables the template creates)	Analytics	Analytics, Basic or Auxiliary.
   Interactive retention (days) (Analytics custom tables the template creates)	The workspace's	4 to 730.
   Total retention (days) (custom tables the template creates)	None set: the table's interactive retention	4 to 4383, and not shorter than the interactive retention.
   Transformation (KQL)	source	Leave it empty to send the data as it is. A custom table's data without TimeGenerated still gets the time of ingestion. Text in AdditionalFields is read as JSON.

4. Click Generate. The drawer shows Before you deploy (the generator's warnings), the Deploy command, the file name with Download and Copy, and the template. When the request is refused, the drawer shows the reason above the button.

A retention value that is not a whole number is marked under its field ("Enter a whole number of days."), and nothing is generated until it is fixed. Changing the type or any option clears the generated template, so the template always matches the fields above it.

What each type produces:

• DCR - The rule as an Azure resource, with the placeholders above for anything left empty. The deploy command first creates each custom table with az monitor log-analytics workspace table create, then the rule with az monitor data-collection rule create --rule-file <name>.json. On the Auxiliary plan, which the Azure CLI can't create, create the table with the ARM or Bicep template first.
• ARM and Bicep - The custom table (for a _CL name, unless Create the custom tables is off) and the rule. The parameters are workspaceName, dataCollectionRuleName, dataCollectionEndpointName (or dataCollectionEndpointId) and location. The outputs are the rule's ID and immutable ID, plus its logs ingestion endpoint for a Direct rule. Deploy with az deployment group create to the workspace's resource group.

Table names start with a letter, then hold letters, digits and underscores, at most 63 characters with _CL included. A lower-case _cl is refused. An Azure table that does not accept the Logs Ingestion API gets a warning. AzureDiagnostics is one of these, so send its data to a custom table such as AzureDiagnostics_CL.

Columns of a custom table:

• id, BilledSize, IsBillable, InvalidTimeGenerated, TenantId, Title, Type, UniqueId and columns whose names start with _ are left out, without a warning.
• A column whose name a custom table cannot hold is left out too, with a warning naming it. A column name starts with a letter and holds 2 to 45 letters, digits and underscores. A workspace's AzureDiagnostics table has several such columns.
• A table holds at most 500 columns.
• AdditionalFields is created as dynamic, as Microsoft Sentinel's tables hold it, even when the schema says string.

Several Schemas at Once

One template can cover several schemas: its rule sends each schema's data to its own table.

1. On the Schemas tab, select 2 to 10 schemas and click Generate template on the bar (see Bulk Actions). One selected schema opens the one-schema drawer described above. Hidden schemas are left out, as their row menu leaves them out. With more than 10 selected, the button is disabled and the bar says "n schemas selected. One template holds at most 10 tables: select 10 or fewer to generate it."

2. The Generate template drawer describes one template for all of them: "its rule sends each schema's data to its own table".

3. Under Microsoft Sentinel, Table name prefix and Table name suffix name the tables. Each table is prefix + schema name + suffix. The default is no prefix and _CL, so each schema's name becomes a custom table: Firewall becomes Firewall_CL, and with the prefix Lake_, Lake_Firewall_CL. Characters a table name cannot hold become _: web-proxy becomes web_proxy_CL. A list under the fields shows every schema and its table.

4. Name each table gives every schema an input of its own, starting from the name the prefix and suffix gave it. The prefix and suffix fields are hidden while it is on; turning it off goes back to them.

5. A name Azure would refuse is marked next to it, and Generate moves to it instead of generating:

   Message	When
   Start with a letter, then use letters, digits and underscores.	An invalid name
   Use at most 63 characters, _CL included.	Too long
   Write _CL in capitals.	_cl
   Another schema's table has this name.	Two tables with the same name, whatever the case

6. The rest of the drawer is the one-schema drawer's: the rule's fields, Generate, and the result. The rule's default name comes from the first table: vmetric-, its name less _CL in lower case, then -dcr. The plan and retention apply to every custom table in the template. Create the custom tables applies to all of them. The stream for each table is Custom- and its name less _CL.

Like the one-schema drawer, this one has two steps, Settings then Template. Generate moves the drawer to Template, which shows Before you deploy (the generator's warnings), the Deploy command, the file with Copy, and the template; Download and Back are in the action bar. Renaming a table or changing a setting after Back clears the template and disables its step until Generate runs again.

The console sends the request to POST /api/library/schema/template with type, schemas: [{ id, tableName }] and the rule options. The public API (/api/v1), PowerShell and MCP generate a template for one schema only.

Grok Patterns

A grok pattern is a reusable extraction pattern that the Grok processor applies to pull structured fields out of unstructured text.

Grok Patterns Table

The table lists each grok pattern with the following columns:

• File Name - The pattern's name. Click to open the detail drawer.
• Description - The pattern's description.
• Assigned resources - Count of resources referencing this pattern.

Above the table, the Search field filters by name, the Origin filter narrows the list to All, Platform, Custom or Hidden patterns, and the Create new Grok pattern button opens the create flow. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage pattern and Clone pattern. As with schemas, the destructive action depends on where the pattern came from: a pattern you created offers Delete pattern, while a platform pattern offers Hide pattern instead, since the platform's own patterns cannot be deleted. A hidden pattern offers Restore pattern, and a platform pattern you have edited also offers Reset to platform, which replaces your changes with the platform's current version under the same name.

Create a Grok Pattern

Click Create new Grok pattern and complete three steps:

Grok Pattern Details

• File name - Required. Between 3 and 64 characters.
• Description - Optional.

Grok Pattern Configuration

Enter the pattern content in a Monaco editor, for example:

%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status}

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Grok Pattern Details

The detail page has three tabs:

• General Overview - Displays and lets you edit the Name and Description, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
• Grok Pattern Configuration - Shows and lets you edit the Grok Pattern Content.
• Activity Logs - A searchable record of actions performed on the pattern.

SNMP Vendors

An SNMP device is identified by the sysObjectID it reports, against a built-in list of about 8,900 devices, and otherwise by its sysDescr. The SNMP Vendors tab shows that list and lets you add a sysObjectID, correct one, or hide one so that devices reporting it are identified by their sysDescr instead.

The tab is available on the Standard and Enterprise editions. Without it, the built-in list still identifies devices, and a device's Vendor can still be chosen by hand.

SNMP Vendors Table

The table lists each sysObjectID with the following columns:

• sysObjectID - The value the device reports. Rows you changed carry a tag: Added, Modified, or Hidden. Built-in rows carry none.
• Vendor - The manufacturer.
• Model - The model, where one is known.
• Device class - What kind of device it is.

Above the table, the search field (Search by sysObjectID, vendor or model) filters the list, the Source filter narrows it to All, Built-in, Added, Modified, or Hidden rows, the Class filter narrows it by device class, and the Create new SNMP vendor button opens the create dialog.

The row menu offers only what applies to the row and what you may do:

Row	Actions	Needs
Built-in, untouched	Edit (creates a change) and Hide	Library create
Added	Edit / Delete	Library edit / Library delete
Modified	Edit and Hide / Reset to built-in	Library edit / Library delete
Hidden	Restore	Library delete

Create or Edit an Entry

Create new SNMP vendor and Edit open the same dialog:

• sysObjectID - Required. Fixed once the entry exists.
• Vendor - Required. At most 64 characters. Records carry it in observer.vendor. A vendor that names a manufacturer the Director has datasets for (Cisco, F5, Fortinet) lets those datasets read the device.
• Device class (optional) - Networking, Computer, Storage, Printer, Phone, Power, KVM, Video, or Software. Records carry it in observer.type.
• Model (optional) - At most 128 characters. Records carry it in observer.product.

The SNMP device's connection check opens the same dialog from Add to SNMP vendors, for a device whose manufacturer is unknown.

Limits and Delivery

• A tenant keeps up to 5,000 changes.
• Changes go at once to the tenant's Directors on v2.2.3 or later as library/snmp_vendor/vendors.json, and a running device picks them up at its next identification.
• A standalone Director reads the same file from config/library/snmp_vendor/vendors.json:

{"entries": [
  {"object_id": ".1.3.6.1.4.1.99999.7", "vendor": "Acme", "device_class": "NETWORKING", "model": "Edge 7"},
  {"object_id": ".1.3.6.1.4.1.9.1.1208", "hidden": true}
]}