Skip to main content

gNMI

Pull

Synopsis​

Director subscribes to one network device's gNMI server: Cisco IOS XR, IOS XE and NX-OS, Juniper, Arista, Nokia, and the others that speak gNMI. It dials the server, holds one Subscribe stream open, and writes what the server streams as ECS records: interface counters every ten seconds, an interface's operational state the moment it changes, BGP sessions, ACL hits, AAA and platform state, and whatever else the device's YANG models expose. Nothing is installed on the network device. A Junos release or setup that streams over Juniper's JTI service rather than gNMI is read by the Juniper JTI device.

One device, one gNMI server. Monitoring forty switches takes forty devices. This is the dial-in mode: Director connects to the server. The dial-out mode, where the network device connects to the collector, is not part of this device: for Cisco IOS XR, IOS XE and NX-OS, it is the Cisco MDT device, and for Nokia SR OS, the gNMI dial-out device.

What to subscribe to is declared by datasets. A dataset is one subscription: a path of the server's data tree, how the server streams it (sampled on an interval, sent on every change, or as the server decides), and the table its values are written to. A device runs any number of datasets, all of them over the one stream. A dataset can instead be a tag subscription: its values are not written as records of their own, and instead label the records of the device's other datasets, such as an interface's description on its counters.

The device dials its server and binds no port, so gNMI devices never collide on one. One Director owns each device: two would subscribe twice and write every value twice.

note

The gNMI device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows gNMI devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.

Schema​

- id: <numeric>
  name: <string>
  description: <string>
  type: gnmi
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    username: <string>
    password: <string>
    encoding: <string>
    tls:
      status: <boolean>
      server_name: <string>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      insecure_skip_verify: <boolean>
      min_tls_version: <string>
      max_tls_version: <string>
    redial: <numeric>
    keepalive_time: <numeric>
    keepalive_timeout: <numeric>
    max_msg_size: <string|numeric>
    updates_only: <boolean>
    depth: <numeric>
    origin: <string>
    prefix: <string>
    target: <string>
    vendor_specific: <string[]>
  definitions: <definition[]>

Configuration​

The following fields are used to define the device:

Device​

FieldRequiredDefaultDescription
idY-Unique numeric identifier
nameY-Device name
descriptionN-Optional description
typeY-Must be gnmi
tagsN-Optional tags
pipelinesN-Optional pre-processor pipelines
statusNtrueEnable/disable the device

Connection​

FieldRequiredDefaultDescription
addressY-Host name or IP address of the gNMI server, without a port. A wildcard address (0.0.0.0, ::) is refused, and so is a host and a port written as one. An IPv6 address may carry a zone (fe80::1%eth0).
portN57400The gNMI server's port, from 1 to 65535. Arista's default is 6030, Juniper's 32767.
usernameN-Sent with password as the username and password metadata of every call, as every vendor's gNMI server reads them. Plain text, ${ENV_VAR}, or $secret{...}.
passwordN-Plain text, ${ENV_VAR}, or $secret{...}. A password with no user name is refused.
encodingNprotoWhat the server is asked to encode values in: proto, json, json_ietf, or bytes. proto keeps every value typed, except from a Nokia SR OS router whose proto-version is v0.7.0: its binary encoding (proto_bytes) is written in base64, with a warning once a stream. Use json for SR OS, or set its proto-version to latest. JSON_IETF writes 64-bit numbers as strings (RFC 7951), and they stay strings (see Values).
tlsNoffHow the connection is secured. See TLS.

Subscribe Stream​

FieldRequiredDefaultDescription
redialN10Seconds the device waits before it dials again after the subscription failed or ended, from 1 to 3600. A duration of whole seconds (30s, 2m) is accepted.
keepalive_timeN600Seconds a connection may be quiet before gRPC pings the server, from 0 to 86400; 0 turns the pings off. It decides how soon a connection whose far end is gone is noticed (see Lost Connections). gRPC sends no ping more often than every ten seconds, and a server refuses pings sent more often than it allows (five minutes, for gRPC's own servers): keep it well above that.
keepalive_timeoutN20Seconds gRPC waits for the ping's answer before it gives the connection up, from 0 to 86400. Not used, with a warning, when keepalive_time is 0.
max_msg_sizeN4MBThe largest message taken from the server: a number of bytes, or a size such as 16MB (binary units), from 4 KiB to 1 GiB. A notification that carries a routing table as JSON can be larger than the default.
updates_onlyNfalseAsks for changes alone: the server does not send the state the subscription starts from.
depthN0The depth extension: the server sends this many levels below a path, from 0 to 65535. 0 leaves the extension out.
vendor_specificN-A list. juniper_header decodes the header Juniper's line cards put on what they stream (see Juniper's Header).

Subscription Prefix​

FieldRequiredDefaultDescription
prefixN-The path every dataset's path goes under. The origin may be written in front of it (openconfig:/components).
originN-The prefix's origin: the YANG model it is in, which every dataset that names no origin of its own is in too.
targetN-The target the server is asked for, for a server that serves several.

With none of the three set, the request carries no prefix at all, rather than an empty one, which some servers refuse.

A configuration error, such as no address, a value out of its range, a value that is not a number or a switch, or an encoding or a vendor option that does not exist, stops the device at start with the connection state Configuration error, and the message names the field. A number is read whole: a fraction, or a ${ENV_VAR} that is not set, is refused, never converted. A switch is true or false, or text that says so; anything else is refused, never read as false.

Connection​

Dialing​

The device dials address:port over gRPC and calls Subscribe, signed in with the username and the password as metadata.

  • The credentials are read at every dial, so a password rotated in the vault is sent from the next dial on, and restarts nothing.
  • Credentials that cannot be read, such as a variable that is not set or a secret store out of reach, fail the dial with the state Authentication error, and nothing is dialed. The message names the field, never its value.
  • gRPC sends metadata of printable ASCII alone, so a username or a password with any other character is refused at the dial, saying so.
warning

A username sent over a connection TLS does not secure is sent in cleartext. The device says so in its log when it starts, and runs. Send credentials without TLS only where the network between Director and the network device is trusted.

TLS​

TLS is on when tls.status is true. The server's certificate is then checked, against tls.ca_name or the system's trust store, unless tls.insecure_skip_verify is on.

FieldRequiredDefaultDescription
statusNfalseSecures the connection
server_nameNthe addressThe name the certificate is checked against
ca_nameNthe system's trust storeThe authority of the server's certificate: PEM inline, a file under the Director's root, ${ENV_VAR}, or $secret{...}
cert_name, key_nameN-A client certificate and its key, for a server that identifies its clients by one. Both or neither.
passphraseN-For an encrypted key
insecure_skip_verifyNfalseSkips the check of the server's certificate
min_tls_version, max_tls_versionNGo'stls1.0, tls1.1, tls1.2, or tls1.3; another value is refused

With tls.status off, the other tls fields are not used, and a device that names a CA or a certificate says so in its log when it starts.

The TLS material is read when the device starts, and material that cannot be read then stops it with Configuration error. The Director reads it again whenever it compares the device's configuration: at every change of the configuration, and every few seconds for a device that names a $secret{...}. Material that changed where it is kept restarts the device. Material that cannot be read at such a check, with the settings unchanged, restarts nothing: the device goes on with what it read, and says so in its log every ten minutes for as long as it lasts.

Lost Connections​

A subscription that fails or ends is asked for again after redial seconds, for as long as the device runs. This covers a server that cannot be reached, refuses the call, or ends the stream, and a connection that is lost.

A connection whose far end is gone without a word, such as a switch that rebooted or a NAT entry that expired, is noticed by gRPC's pings. With nothing from the server for keepalive_time seconds, the server is pinged, and the connection is given up when no answer comes within keepalive_timeout. Keepalive is on by default: with the pings off, only the operating system's TCP keepalive notices a dead connection, after two hours of silence on Linux and on Windows, during which the device reads as connected.

Subscribing​

Every dataset is one subscription of one request: a STREAM subscription list in the device's encoding, under its prefix, with updates_only and the depth extension when they are set. The tag subscriptions come first, so a server that sends the state of its paths in the order asked sends the labels before the records they go on.

The device is Connected once the server has answered: its first notification, or the sync_response that says it has sent the state of every path. A server that streams a path on change sends nothing until something changes, and the device stays connected.

Datasets​

The device's definitions are its datasets. In the console they are the datasets and profiles assigned to the device in its Data collection; see Datasets and Profiles. In a configuration file each dataset is a definition named gnmi_subscription_collector, the device's one dataset type:

DatasetDefinitionDefinition IDKindOne record per
subscriptiongnmi_subscription_collector18150metricnotification and list entry
  • Any number of datasets, all over the one stream, subscribing to 512 paths at most.
  • Pipelines. A dataset's records enter the pre-processing pipeline the dataset selects, or the device's when it selects none.
  • Nothing assigned is allowed. The device then asks its server for its capabilities every minute, signed in as it is, and writes nothing, so its connection state follows the server. The call is given ten seconds.
  • A dataset that cannot be read is left out, with a warning in the device's log that names it: a table or a path that is not valid, a mode that does not exist, or a value out of its range. So is a dataset that subscribes to the same path as an earlier one, with the same origin: a server answers a path asked for twice twice, or refuses the request. The rest of the device runs. With every dataset assigned to it left out, the device stops with Configuration error.

Dataset Properties​

FieldRequiredDefaultDescription
tableY-Names the record's values: they sit under gnmi.subscription.<table>. Letters, digits, and underscores, at most 64, not starting with a digit, and none of table, origin, path, target, labels, or deleted. Two datasets may share one.
pathY-The path subscribed to, as gNMI's path strings write one: /interfaces/interface[name=Ethernet1]/state. The leading slash is optional. A list entry's keys follow its element in square brackets; a value may hold a slash, and a backslash escapes a closing bracket or a backslash. * stands for any one element and ... for any number, and a key whose value is * picks every entry. The origin may be written in front of it (openconfig:/interfaces).
originNthe device'sThe YANG model the path is in, for a server that asks for one (openconfig, rfc7951, or a vendor's). Named once: refused when the path names another.
subscription_modeNsamplesample (on the interval), on_change (on every change), or target_defined (the server decides, leaf by leaf)
intervalN10Seconds between samples, from 1 to 86400, sent for the modes that sample. A path streamed on change has no interval.
suppress_redundantNfalseAsks a sampling server to send only what changed
heartbeat_intervalN0Seconds after which the server sends a value all the same, changed or not, from 0 to 86400. 0 sends none.
emit_deletesNfalseWrites a record of every path the server says was deleted (see Deletions)
aliasesN-Further paths whose values are the dataset's, each with its origin in front when it names one; at most 16. For a server that answers a subscription with paths other than the one it was asked for: Juniper's line card sensors come back under /components.
tag_subscriptionNfalseMakes the dataset a tag subscription (see Tag Subscriptions)
tag_matchNname, or elements when tag_elements names someHow a tag subscription's values find the records they label: name, elements, or unconditional
tag_elementsN-For elements: the path elements, by name, whose keys have to be the same (interface, subinterface); at most 16
labelsN-An object of names and values written on every record of the dataset as ECS labels. At most 32; a name is letters, digits, underscores, and hyphens (at most 64, not starting with a digit or a hyphen), and a value is text, a number, or a switch of at most 256 characters.
pipeline_nameNthe device'sThe pre-processing pipeline the dataset's records enter

From a Notification to Records​

Which Dataset​

An update belongs to the dataset whose path, or alias, accounts for the most of its path: an update under /interfaces/interface/state/counters belongs to a dataset of that path rather than to one of /interfaces/interface/state. Paths are compared by their elements' names, without keys and without the namespace a server may write in front of one (openconfig-interfaces:interfaces is interfaces). Among datasets that account for as many elements, the one of the update's origin wins; among those, the earliest. An update no dataset claims is left out and logged, once per path for as long as it lasts.

A Record per List Entry​

The values of one notification are written as one record per dataset and per list entry: the counters of Ethernet1 and of Ethernet2, sent in one notification, are two records. The entry's keys are the record's labels, so an interface's name is gnmi.subscription.labels.name. A key whose name an entry above already took is named after its element too: /network-instances/network-instance[name=default]/protocols/protocol[name=BGP] gives name and protocol_name. Every hyphen in a label's name is an underscore.

Field Names​

A value is written under the dataset's table, named after the elements of its path below the dataset's, each a level of its own, without namespaces and with an underscore for every hyphen. in-octets, under a subscription to /interfaces/interface/state, is gnmi.subscription.<table>.counters.in_octets. A value at the dataset's path itself is named after the path's last element; an object there, such as a JSON value of a container, is the record's values.

Values​

Values are written as the server types them: text, whole numbers (including unsigned ones larger than a signed 64-bit integer holds), numbers, switches, and lists of them.

  • A decimal is written as the number it stands for.
  • Four bytes are a 32-bit float, as OpenConfig's ieeefloat32 is. Other bytes, and a protobuf message of the server's own, are written in base64. A proto_bytes value is warned of once a stream.
  • A JSON or a JSON_IETF value is written as the object or the list it encodes, its members named as fields are. JSON_IETF writes 64-bit integers and decimals as strings (RFC 7951), and they stay strings: without the YANG model there is no telling them from text. Use proto to keep them typed.
  • A NaN or an infinity, which JSON cannot hold, and a google.protobuf.Any, which is not decoded, are left out of the record and logged, once per path for as long as it lasts.

A record is dated at its notification's timestamp. A notification dated before 2000, such as zero or a server that sends milliseconds or seconds where the protocol says nanoseconds, is dated when it arrived.

Tag Subscriptions​

A tag subscription's values are not written as records of their own. They are kept by the name of their leaf, and label the records of the device's other datasets, from the notification that brought them on. A label is named after the tag subscription's table and the leaf: the description of an interface, from a tag subscription of table descr, is gnmi.subscription.labels.descr_description. An object's leaves are labels of their own, and a leaf-list is its values joined by commas.

How a value finds its records depends on tag_match:

  • name: the records whose path has a list entry with the same name key. The description of the interface named Ethernet1 goes on every record of an entry named Ethernet1.
  • elements: the records whose path has entries of every element named in tag_elements, with the same keys. [interface, subinterface] matches Ethernet1's subinterface 100 and no other.
  • unconditional: every record of the device.

A value sent empty takes its label away, and so does the deletion of its leaf. Labels are kept for as long as the device runs, across a lost connection, since a server dialed again sends the state of its paths again.

Deletions​

A dataset with emit_deletes writes a record of every path within its own that the server says was deleted: a deletion event with the path deleted, keys included, under gnmi.subscription.deleted, and its keys as labels. A deleted path above a dataset's, such as an interface removed where a dataset subscribes to its counters, is written by every dataset below it that asks for deletions. A dataset that does not ask writes none.

Juniper's Header​

With vendor_specific: [juniper_header], the header Juniper's line cards put on a response gives its records the labels component (when it names one), component_id, and sub_component_id. A header that cannot be read is logged, and its records go without it.

Records​

Every record is one ECS 8.11 document. These fields are on every record:

FieldValue
@timestampThe notification's time (see Values)
ecs.version8.11.0
event.modulegnmi
event.datasetgnmi.subscription
observer.nameThe device's name in the console
observer.hostnameThe Director's host name
observer.vendor, observer.productVirtualMetric, gNMI
server.address, server.portThe network device's address and port, as configured
server.ipThe address, when it is an IP address
network.protocol, network.transportgnmi, tcp
labels.<name>The dataset's labels, when it has any
gnmi.subscription.tableThe dataset's table
gnmi.subscription.pathThe path that claimed the values, the dataset's or an alias, under the device's prefix, without keys
gnmi.subscription.originIts origin, when it has one
gnmi.subscription.targetThe target the server named, when it named one
gnmi.subscription.labels.<name>The keys of the list entries on the path, and the labels of the tag subscriptions and Juniper's header

A record of values, what a notification sent for one list entry, also has:

FieldValue
event.kindmetric
event.category, event.typehost, info
gnmi.subscription.<table>.<field>The values (see Field Names)

A deletion, for a dataset that asks for them, has instead:

FieldValue
event.kindevent
event.category, event.typeconfiguration, deletion
event.actiondelete
gnmi.subscription.deletedThe path deleted, with its keys

Device State​

What happenedState
The server answered the subscription, or, with no dataset, the capabilities callConnected
The server cannot be reached, the call ran out of time, or the stream was ended or lostConnection error
The TLS handshake failed: a certificate that does not verify, or a server that does not speak TLSCertificate error
The server refused the credentials (Unauthenticated, PermissionDenied), or the credentials cannot be readAuthentication error
The server refused the subscription (InvalidArgument, Unimplemented, NotFound, FailedPrecondition, OutOfRange) because it does not support an encoding, a path, or a mode; or a message was larger than max_msg_sizeConfiguration error
Any other statusError

A failure is logged once for as long as it lasts, with what to check, and its end is logged. Records that could not be written do not change the device's state: the server answered.

Configuration Changes​

Any change of the device, or of a dataset assigned to it, restarts the device, which subscribes anew. So does an edit that no longer validates, which the restart then reports as Configuration error. Credentials are compared as they are configured, which is a secret's name and not its value, so a secret rotated in its store is sent from the next dial on and restarts nothing. TLS material is compared as described in TLS.

When the device stops, it ends the subscription at once, then flushes what it has queued before it is stopped, so a restarted device never has its predecessor still writing beside it.

note

The gNMI device is not simulated. A dataset's records are a network device's own data tree, a shape a generated sample cannot stand in for.

note

The device reads values encoded as JSON_IETF containers. It also reads the path shapes the vendors document: Cisco's prefix per entry, Arista's whole paths, and Juniper's paths under /components with its header.

Console​

In the console, go to Fleet Management > Devices > gNMI, listed under Protocols beside Modbus, Prometheus, and Siemens S7, and click Add new device. The wizard's steps are:

  1. General settings, as on every device.

  2. Connection settings:

    • Address, Port, and Encoding.
    • Authentication: Username (Optional) and Password (Optional).
    • TLS settings: the TLS encryption switch. Enabled, it shows Skip certificate verification, the CA certificate, Server name (Optional), the minimum and maximum TLS version, and a Client certificate section with Certificate (Optional), Private key (Optional), and Passphrase (Optional).

    While a username is set and TLS encryption is disabled, the step shows the warning The credentials are sent in cleartext.

  3. Advanced settings:

    • Subscribe stream: Redial delay, Maximum message size (Optional), Keepalive time, Keepalive timeout, Depth, Updates only, and Juniper header.
    • Subscription prefix: Prefix (Optional), Origin (Optional), and Target (Optional).

  4. Data collection: the gNMI datasets the device subscribes to.

  5. Review and create.

The device's page has the same tabs, and Data collection.

A gNMI dataset is created in Fleet Management > Datasets as the type gNMI Subscriptions; see Management. Its settings are Table name, Path, Origin (Optional), Subscription mode (Sample, On change, or Target defined), Sample interval (Optional), Suppress redundant, Heartbeat interval (Optional), Deletion records, Aliases (Optional), Tag subscription with Tag match (By name, By elements, or Every record) and Tag elements, the labels, and the pre-processing pipeline.

A setting the chosen mode hides is neither checked nor shown. Sample interval (Optional) and Suppress redundant show only for the modes they apply to, so a dataset streamed On change has no sample interval. Tag match shows only for a tag subscription, and Tag elements only when Tag match is By elements.

Examples​

Arista Switch, Interfaces and BGP​

Subscribing to an Arista switch's interface counters every 10 seconds, its interfaces' operational state and BGP sessions on every change, and its interfaces' descriptions as labels on the other records...

- id: 612400301
  name: core-sw-01
  description: Arista core switch, interfaces and BGP
  type: gnmi
  status: true
  properties:
    address: "192.0.2.31"
    port: 6030
    username: telemetry
    password: ${GNMI_PASSWORD}
    encoding: proto
    tls:
      status: true
      ca_name: ops-ca.pem
      server_name: core-sw-01.example.com
  definitions:
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: ifcounters
        origin: openconfig
        path: /interfaces/interface/state/counters
        subscription_mode: sample
        interval: 10
        labels:
          role: core
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: ifstate
        origin: openconfig
        path: /interfaces/interface/state/oper-status
        subscription_mode: on_change
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: descr
        origin: openconfig
        path: /interfaces/interface/state/description
        subscription_mode: on_change
        tag_subscription: true
        tag_match: name
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: bgp_neighbors
        origin: openconfig
        path: /network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-state
        subscription_mode: on_change
        emit_deletes: true

Cisco IOS XR Router Under a Prefix​

Asking an IOS XR router for JSON_IETF under one prefix, which every dataset's path goes under, with a longer keepalive and a larger message size...

- id: 612400302
  name: edge-rtr-01
  description: IOS XR edge router, platform state
  type: gnmi
  status: true
  properties:
    address: "edge-rtr-01.example.com"
    username: telemetry
    password: ${GNMI_PASSWORD}
    encoding: json_ietf
    origin: openconfig
    prefix: /components
    tls:
      status: true
      ca_name: ops-ca.pem
    keepalive_time: 900
    keepalive_timeout: 30
    max_msg_size: 16MB
  definitions:
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: cpu
        path: /component/cpu/utilization/state
        interval: 30

Juniper MX Line Card Sensors​

Subscribing to a Juniper MX's line card memory, which comes back under /components and is claimed by an alias, with Juniper's header decoded into labels...

- id: 612400303
  name: mx-01
  description: Juniper MX, line card memory
  type: gnmi
  status: true
  properties:
    address: "192.0.2.33"
    port: 32767
    username: telemetry
    password: ${GNMI_PASSWORD}
    tls:
      status: true
      ca_name: ops-ca.pem
    vendor_specific:
      - juniper_header
    depth: 3
  definitions:
    - name: gnmi_subscription_collector
      status: true
      properties:
        table: linecard_memory
        path: /junos/system/linecard/cpu/memory
        interval: 60
        aliases:
          - /components