Commvault Datasets

Commvault datasets define what a Commvault device reads from a Commvault CommServe (dataset type commvault). They attach to devices of type commvault, and in Datasets & Profiles the category is labelled Commvault. To create and assign datasets through the web interface, see Management. For the dataset and profile model, see Overview.

Collector Types

There are four datasets, one per data model, each with its own definition id, event.dataset, interval, checkpoint, and pipeline, so it can be searched, routed, and processed on its own. Each is a tile in the wizard's Commvault category.

Dataset (console title)	Dataset	Definition	event.dataset	Definition id	Kind	Default interval	Minimum
Commvault Clients	client	commvault_client_collector	commvault.client	5100	inventory	3,600 s	60 s
Commvault Alert Rules	alert_rule	commvault_alert_rule_collector	commvault.alert_rule	60	inventory	3,600 s	60 s
Commvault Jobs	job	commvault_job_collector	commvault.job	62	event	60 s	10 s
Commvault Events	event	commvault_event_collector	commvault.event	61	event	60 s	10 s

Every dataset takes these properties:

Field	Required	Type	Default	Description
interval	N	int	per dataset	Seconds between readings. At least the dataset's minimum above.
pipeline_name	N	string	the device's	The preprocessing pipeline the dataset's records enter.
initial_lookback	N	int	the device's	Jobs and events only. Seconds before its first reading where the dataset starts when it has no checkpoint.

The clients, jobs, and events datasets also take the filter properties under Filters.

Wizard fields

Every Commvault dataset's Dataset configuration step says how the dataset is read and has an Interval (Optional) field, in seconds: left empty, the dataset is read at its default; otherwise at least its minimum. The clients, jobs, and events datasets also show a Filters section (see Filters).

Rules

• Intervals. An interval under the dataset's minimum is left out with a warning in the device's log, and the rest of the device runs.
• One of each. A dataset type is read once per device, however many datasets of it are assigned. The first decides its settings; the others are left out with a warning.
• Pipelines. A dataset's records enter the pipeline the dataset selects, otherwise the device's. Two datasets on one device can reach two pipelines.

Inventories

The clients and alert rules datasets write every item at every reading. From a dataset's second reading on, each record carries event.action:

event.action	event.type	Meaning
added	creation	Not in the last delivered reading
changed	change	Its fields differ from the last delivered reading's
unchanged	info	As it was
removed	deletion	Gone: written once, as it was last seen, dated now

• The baseline is the last delivered reading. A reading that fails, or whose records could not all be written, never becomes the one the next compares with, so nothing is marked removed because a request failed.
• A client whose properties do not answer is left out of that reading and is not gone: what the last reading held of it is kept for the next to compare with.
• The first reading after a start marks nothing: the baseline is kept in memory.

Commvault Clients

Reads every client of the CommServe — the CommServe itself, MediaAgents, file servers, and virtualization and cloud pseudo-clients — with its operating system, version, installed agents, and client groups. A client is known by its id. The client rule, when the dataset has one, runs on the client list before any client's properties are asked for, so a client it leaves out costs no request.

Dataset	One record per	Fields
client	Client of the CommCell	host.name (the client name), host.hostname, host.id (the client GUID), host.os.name, host.architecture; commvault.client.id, name, display_name, host_name, guid, commcell_name, description, install_directory, cvd_port, evmgrc_port, os.name/type/subtype/processor_type, version, release, time_zone, agents (installed agents, sorted), groups (client groups, sorted), commserve, media_agent, virtual, deleted, infrastructure, readiness (not compared)

readiness is written but not compared: a client going offline is how it is doing, not what it is.

Commvault Alert Rules

Reads each alert rule configured on the CommServe: its category, type, status, and notification. Every rule is written at every reading, marked like the clients, so a rule edited or deleted is seen. A rule is known by its alert id. The alert rules name no client and take no filter.

Dataset	One record per	Fields
alert_rule	Alert rule	rule.id, rule.name, rule.description, rule.category, rule.author; commvault.alert_rule.id, name, description, category.id/name, type.id/name, creator.id/name, notification_type, status, organization_id, created

Jobs and Events

The jobs and events datasets read what happened since their checkpoint, and resume from it across a restart or a failover. The first reading of a dataset with no checkpoint goes back as far as the initial lookback, a day unless the dataset or the device says otherwise. See Commvault for how checkpoints are kept.

Commvault Jobs

Reads backup and restore jobs, active and finished, across every client: their status, phase, level, sizes, files, client, subclient, and storage policy. A job is written each time the CommServe updates it — when it is first seen, as it moves on, and when it ends — so its final state arrives. Records are written in the order the jobs were updated.

Dataset	One record per	Fields
job	Update of a job	host.name (the client the job ran for), user.name, event.start, event.end, event.duration (ns), event.type (end once the job has ended, else info), event.outcome; commvault.job.id, type, operation, status, localized_status, pending_reason, phase.id/name, backup_level.id/name, percent_complete, start, end, last_update, elapsed (s), size.application/media_on_disk/percent_savings, files.total/failed/failed_folders, commcell_id, client.id/name/display_name, subclient.id/name/backup_set_id/backup_set_name/instance_id/application_id/application, storage_policy.id/name, user.id/name

event.outcome is success once a job has Completed, and failure once it has Failed, Failed to Start, or been Killed. It is unset for any other status.

Commvault Events

Reads the CommServe's events: their code, severity, subsystem, and client. Events are written in the order they happened.

Dataset	One record per	Fields
event	CommServe event	message (the description), event.code (for example 14:196), event.severity (the CommServe's own number), event.provider (the subsystem), host.name (the client); commvault.event.id, code, code_string, subsystem, description, severity, acknowledged, type, job_id, client.id/name/display_name

Filters

By default the clients, jobs, and events datasets read every client: its properties, its jobs, and its events. A client rule narrows that, the jobs dataset can list some job types only, and the events dataset can read the events from a severity up. A dataset that sets none of these reads what it always read. The alert rules take no filter.

For how the Filters section works in the console, see Collection Filters.

Client rule

The client rule is the client scope of the dataset's filter, filter.client. In the console it is the Clients group, on the clients, jobs, and events datasets.

Field	Console label	Description
filter.client.include	Include clients	Patterns on a client's names: its client name, display name, and host name. A client is read when one of them matches. A pattern that is a bare number, such as 7, also names the client with that id.
filter.client.exclude	Exclude clients	Patterns on the same names. A client one of them matches is left out, even when include or groups names it.
filter.client.groups	Client groups	Patterns on client group names. When there are any, a client is read only when it also belongs to a group one of them matches.

• Patterns ignore case, * matches any run of characters and ? one, and a pattern matches a whole name, not a part of it: sql* keeps sql01 but not mssql01. A wildcard matches names only, so *01 does not keep client 101 by its id.
• Groups need a permission. Using groups needs the Commvault user to have the permission to view client groups. A request for them the CommServe refuses fails the reading, which keeps its checkpoint or its baseline and is read again. Patterns that match no group keep no client, and the device's log says so.
• Group membership is refreshed at every reading of the clients dataset, and every ten minutes for jobs and events, so a client that joins a group has its jobs and events written within ten minutes.
• Jobs are judged by the client their record names (commvault.job.client). A job that names no client fails include and groups, and passes exclude.
• Events are judged by the client they name (commvault.event.client). An event that names no client is always written: a client rule says nothing about it.

Job types

Field	Console label	Default	Description
job_types	Job types	every job type	The jobs dataset only. Backup, Restore, AUXCOPY (Auxiliary copy), or WORKFLOW (Workflow). The types are sent to the CommServe, which lists only those.

Minimum severity

Field	Console label	Default	Description
min_severity	Minimum severity	information	The events dataset only. information, minor, major, or critical: the least grave events read. information reads every event. The CommServe is asked for that severity and above only.

job_types and min_severity sit next to interval in the definition's properties.

Following the Clients dataset

In the console, a jobs or events dataset can use a Commvault Clients dataset's client rule rather than keep a copy of it: in its Filters section, choose Use the filter of another dataset and pick the clients dataset. Changes to that dataset's rule apply here too. job_types and min_severity belong to their own dataset and are never followed.

It is the rule that is followed, not the clients dataset's last reading: the jobs and events judge each record by the client it names, so a client created since the last client reading is judged at once by its names. A rule that uses groups judges it by the group membership the dataset last read, which is refreshed every ten minutes.

When the rule changes

• A client that stops passing the rule (renamed, or taken out of a group) is written once more as removed; one that starts passing it is added.
• A changed filter restarts the device. The inventories' first reading after it marks nothing, rather than marking every client the new rule leaves out as removed.
• Checkpoints are kept per dataset, not per rule, so widening a rule does not go back for the jobs and events the narrower one left out.
• What the rule leaves out still moves the checkpoint on. A job update or an event the client rule leaves out is not written, but the checkpoint moves past it.

Examples

Reading the clients in the SQL Servers group, except test machines, with their backup and restore jobs, and the CommServe's major and critical events...

- id: 512300118 name: cs01 type: commvault status: true properties: address: "commserve.example.com" username: "svc-virtualmetric" password: "$secret{store=corp-vault,ref=backup/commvault}" definitions: - name: commvault_client_collector properties: filter: client: groups: ["SQL Servers"] exclude: ["*-test"] - name: commvault_job_collector properties: job_types: ["Backup", "Restore"] filter: client: groups: ["SQL Servers"] exclude: ["*-test"] - name: commvault_event_collector properties: min_severity: major