Skip to main content
Version: 1.3.0

Microsoft Sentinel Integration

VirtualMetric Director supports Microsoft Sentinel integration through two different approaches: automatic discovery and manual configuration. Choose the method that best fits your environment and requirements.

Prerequisites for both approaches:

  • an Azure subscription with permissions to create resources
  • a Log Analytics workspace required by Microsoft Sentinel

VirtualMetric Director provides an autodiscovery feature for Microsoft Sentinel integration. This enables automatic detection and configuration of Data Collection Rules (DCRs) and their associated streams, simplifying the setup process and providing dynamic updates as your Sentinel environment changes.

Open a terminal with Administrative access and navigate to <vm_root>. Then, type the following command and press Enter.

C:\vmetric-director -sentinel -autodiscovery

Follow the on-screen prompts to complete the setup process. For detailed step-by-step instructions, refer to Microsoft Sentinel Overview.

Required Permissions​

Director needs the following permissions for Microsoft Sentinel integration.

note

If you used the Automation tool with App Registration, these permissions are already configured.

Director needs the following permissions to fetch the Data Collection Rules and their associated streams.

For Data Collection​

For each DCR name prefixed with vmetric:

  1. Navigate to the DCR in Azure Portal
  2. Go to Access Control (IAM)
  3. Select Add > Add role assignment
  4. Assign the following permissions:
    RoleAssignee
    Monitoring Metrics PublisherYour Managed Identity or Application

For Autodiscovery​

To enable the DCR autodiscovery features:

  1. Navigate to the Resource Group containing your DCRs
  2. Go to Access Control (IAM)
  3. Select Add > Add role assignment
  4. Assign the following permissions: |Role|Assignee| |---|---| |Monitoring Reader|Your Managed Identity or Application|
important

The Monitoring Reader role should be assigned at the Resource Group level only. Assigning this role at the Subscription level is not recommended since it is not required for the functionality to work, and it increases the autodiscovery scan duration.

Verification​

After assigning the permissions:

  1. Wait a few minutes for Azure RBAC to propagate
  2. Test the connection using Director
  3. Check the logs for any permission-related errors
tip

If you encounter permission issues, verify that all role assignments are properly configured, that Azure RBAC changes have propagated (can take up to 30 minutes), and that the identity has the correct access scope.

How It Works​

Resource ID-Based Discovery​

Instead of manually configuring the Data Collection Endpoint (DCE) URL, you can provide the DCE Resource ID. For example:

/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Insights/dataCollectionEndpoints/<dce-name>

When using a Resource ID, Director will discover all DCRs associated with the specified DCE, and collect detailed stream information including table names, table schemas (column definitions), and stream configurations.

Caching Mechanism​

The default cache duration is 5 minutes. The cache is automatically invalidated when the configuration file (sentinel.yml) is modified or the cache timeout is reached.

Dynamic Updates​

The autodiscovery feature continuously adapts to changes in your Sentinel environment thanks to which new DCRs are automatically detected, and changes to table schemas are recognized while custom tables and columns are discovered and integrated.

Phantom Field Prevention​

Microsoft Sentinel has moved to DCR-based log ingestion and manual schema management. This change, while powerful, can lead to phantom fields, i.e. data fields that are ingested and billed even though they are not part of the table schema and therefore are inaccessible for querying, and yet are still incurring storage costs.

For a comprehensive understanding of phantom fields, see Sentinel Phantom Fields by ManagedSentinel.

Among the common scenarios that cause these are log splitting with mismatched schemas, temporary fields in transformations, duplicate fields emerging from improper field mapping, and schema modifications without proper cleanup.

Director's autodiscovery feature includes a built-in phantom field prevention mechanism based on the following:

Schema Validation - Automatically discovers table schemas from DCRs, validates each field against the known schema, and discards fields not present in the table schema.

Dynamic Field Mapping - Fields that exist in the schema or are required are kept while others are discarded.

Cost Optimization - Prevents unnecessary data ingestion thereby reducing storage costs while maintaining data accessibility.

The guiding principles for autodiscovery include schema management (regularly review table schemas, update schemas when adding new fields, use autodiscovery to validate field usage), field mapping (let autodiscovery handle field validation, define critical fields explicitly when needed, and monitor for any dropped fields in logs), and cost monitoring (track ingestion volumes, monitor field usage patterns, and verify data accessibility).

The most salient reason for preventing phantom fields is to reduce their impact on cost. Some environments show up to 65% of table data as phantom fields.

Configuration Examples​

Basic Autodiscovery Configuration​

targets:
  - name: sentinel
    type: sentinel
    properties:
      tenant_id: "<your-tenant-id>"
      client_id: "<your-client-id>"
      client_secret: "<your-client-secret>"
      endpoint: "/subscriptions/.../dataCollectionEndpoints/<your-dce-name>"  # Use Resource ID

Filtered Stream Configuration​

You can filter the autodiscovered streams that you intend to use:

targets:
  - name: sentinel
    type: sentinel
    properties:
      tenant_id: "<your-tenant-id>"
      client_id: "<your-client-id>"
      client_secret: "<your-client-secret>"
      endpoint: "/subscriptions/.../dataCollectionEndpoints/<your-dce-name>"
      streams:
        - name: "Custom-WindowsEvent"
        - name: "Custom-SecurityEvent"

Cache Configuration​

You can optionally adjust the cache timeout (in seconds):

targets:
  - name: sentinel
    type: sentinel
    properties:
      endpoint: "/subscriptions/.../dataCollectionEndpoints/<your-dce-name>"
      cache:
        timeout: 300