VirtualMetric DataStream
VirtualMetric DataStream is an automation engine that streamlines data collection and routing for Microsoft Sentinel and other Azure data services. It uses pipelines to process, enrich, and direct data flows to their destinations, simplifying deployment and routing the data to Azure services based on cost and functionality requirements.
Telemetry Pipelines
A telemetry pipeline is a comprehensive end-to-end system that manages the entire journey of log data—collecting, processing, and routing to various endpoints—from source to destination. It is responsible for ensuring that the right information reaches the right destination at the right time.
Users can design the pipeline so as to route the data based on data type, source, or other criteria. Each piece of information that enters the pipeline undergoes several crucial transformations.
Collection
The pipeline first gathers raw data from various sources like network devices and firewalls, operating systems and applications, security tools and monitoring systems, and cloud services and infrastructure.
Processing
The pipeline then processes the data to ensure consistency and quality using various stages that normalize the data to standard formats, parses unstructured data into structured fields, validates the data points to ensure quality and completeness, and enriches it with additional context and metadata.
Routing
The processed data is then directed to the appropriate destinations which involves determining the appropriate destinations based on the type and purpose of use of the data, managing the flow rates and batching.
Enterprise Challenges
In enterprise environments, pipelines are essential due to the need to handle massive volumes of data. One of the challenges they face is managing the diverse types of log data generated by their systems and applications.
The data must be directed to the appropriate destinations for security monitoring, threat detection, compliance, and analysis efficiently and accurately, enabling real-time flow management, with consistent processing and formatting, maintaining data integrity throughout the journey, and enabling complicated routing decisions.
DataStream's routing capabilities enable organizations to direct different types of log data to the most appropriate Azure services based on their security, operational, and compliance requirements:
Security
These logs contain critical security events that require real-time monitoring and immediate alerting. DataStream routes these logs directly to Microsoft Sentinel for threat detection and incident response.
Historical security data is essential for threat hunting, incident investigation, and long-term trend analysis. DataStream routes the data to Azure Data Explorer for efficient querying and analysis.
Compliance
Regulatory compliance data must be stored securely for long-term retention and audit purposes. DataStream routes these logs to Azure Blob Storage for cost-effective storage and comprehensive audit trails.
This approach provides several benefits like optimizing the costs with the appropriate storage solutions, improving query performance for different use cases, enabling flexible retention policies, and maintaining comprehensive security coverage.