Red Hat OpenShift

Pull

Synopsis

Director polls one OpenShift cluster through its API server, with a bearer token, and writes what it reads as ECS documents: the cluster's nodes, projects, and pods, the replication controllers of its deployment configs, its services and service accounts, its persistent volume claims and CSI drivers, its network configuration, its users, and its events. Nothing is installed in the cluster.

What the device reads is decided by the datasets and profiles assigned to it, as for Windows, Linux, and SNMP devices. Each data model is a dataset of its own — see Red Hat OpenShift Datasets.

The cluster is the device. One device reads every node and every project of one cluster.

Nothing it finds is registered as a device. The nodes, projects, and pods the device discovers are records of the cluster's, and they tag the records of the objects inside them (see Tags). None of them becomes a device of its own.

The device dials the API server and binds no port, so any number of Red Hat OpenShift devices may use port 6443. One Director polls each device: two Directors never read the same events twice.

note

The Red Hat OpenShift device needs Director 2.2.3 or later. With no Director or cluster at that version selected, the wizard shows Red Hat OpenShift devices are unavailable on directors or clusters below version 2.2.3 and the device cannot be created.

What the Linux Agent Reads Instead

The device has no metric dataset. Usage, and the pods' logs, are read on every node by the Linux agent, and the device does not read them again:

• Node, pod, and container usage, and their readiness and restarts as metric series over time, come from the Linux agent's Kubernetes Metrics dataset, which also counts the cluster's workloads, services, and namespaces. The Red Hat OpenShift node and pod inventories carry a node's readiness and a pod's phase and restarts too, but as fields of each object's record at every reading, not as metrics.
• The pods' logs come from the Linux agent's Kubernetes Pod Logs dataset.

The Red Hat OpenShift device reads what those datasets do not: the inventory of what the cluster holds, OpenShift's own projects, users, and network configuration, and the cluster's events.

Account

Every request sends the device's token as Authorization: Bearer <token>. The device reads; it changes nothing, and never lists a secret.

A service account with the cluster-reader cluster role reads every dataset:

oc create serviceaccount virtualmetric -n virtualmetric
oc adm policy add-cluster-role-to-user cluster-reader -z virtualmetric -n virtualmetric
oc create token virtualmetric -n virtualmetric --duration=8760h

• oc create token gives a token that expires, and the API server may cap its duration. For a long-lived token, create a kubernetes.io/service-account-token secret annotated with the account's name (kubernetes.io/service-account.name: virtualmetric); the cluster fills in its token.
• A user's OAuth access token (sha256~...) works too, until it expires, a day by default.

A dataset the account may not read keeps the device Connected, and the device's message names each such dataset, and why, until that dataset reads again:

OpenShift poller connected to ocp01. openshift_user_collector reads nothing: the API server refused the account /apis/user.openshift.io/v1/users (403 Forbidden)

The same holds for an API the cluster does not serve, such as project.openshift.io, user.openshift.io, or config.openshift.io on a Kubernetes cluster that is not OpenShift.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: openshift
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    token: <string>
    port: <numeric>
    timeout: <numeric>
    initial_lookback: <numeric>
    tls:
      status: <boolean>
      insecure_skip_verify: <boolean>
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be openshift
tags	N	-	Optional tags
pipelines	N	-	Optional pre-processor pipelines
status	N	true	Enable/disable the device

Connection

Field	Required	Default	Description
address	Y	-	Host name or IP address of the cluster's API server (api.<cluster>.<domain>), or of the load balancer in front of it. A wildcard address, a URL, or host:port is refused. An IPv6 address may be written in one pair of brackets.
port	N	6443, or 80 without TLS	The API server's port
token	Y	-	The bearer token every request sends. See Account.

The token takes plain text, ${ENV_VAR}, or $secret{...} ($secret{id=...} or $secret{store=...,ref=...}). It is resolved at every request, so a token rotated in a vault is sent once the vault's cache lets it through. The configuration keeps the reference, and a changed reference restarts the device.

TLS

Field	Required	Default	Description
tls.status	N	true	The API is called over HTTPS. false calls it over plain HTTP, for an endpoint behind a proxy that terminates TLS.
tls.insecure_skip_verify	N	false	true accepts any certificate. Pin the CA instead where you can.
tls.ca_name	N	the system trust store	A CA to verify the API server's certificate against
tls.server_name	N	address	The name the certificate is verified for, when it is not address
tls.min_tls_version	N	tls1.2	tls1.0, tls1.1, tls1.2, or tls1.3
tls.max_tls_version	N	-	No cap

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Pin the cluster's CA. An API server's certificate is signed by the cluster's own CA unless it was given one from a public CA. Set tls.ca_name to the CA oc verifies it with — the certificate-authority-data of your kubeconfig, decoded from base64 — rather than turning verification off.

warning

With tls.status set to false, the bearer token is sent in cleartext with every request, and anyone who reads it can call the API server as the account until the token expires or is revoked. Use plain HTTP only where the network between the Director and the proxy that terminates TLS is trusted.

Datasets

The device's definitions are its datasets, each named after its type, openshift_<dataset>_collector. In the console they are the datasets and profiles assigned to the device. For the dataset types, their intervals, and their fields, see Red Hat OpenShift Datasets.

Advanced Configuration

Field	Required	Default	Description
timeout	N	60	Seconds each request may take, from 1 to 300. A list is read a page of 500 objects at a time, each page its own request.
initial_lookback	N	86400	Seconds before its first reading where the event dataset starts when it has no checkpoint. 0 starts at the first reading. The dataset's own initial_lookback wins. The cluster keeps an event three hours after it last happened by default, so a day reads every event it still holds.

Connection States

• A configuration error — no address, no token, a reference that does not resolve, a CA that cannot be read — stops the device with Configuration error. The message names the property but never its value.
• At start the device reads who its token is (as oc whoami does) and the cluster's version. A token the server refuses (401) sets Authentication error and stops the device; the Director starts it again later.
• A server that does not answer, or a gateway in front of it answering 502, 503, or 504, sets Connection error; one whose certificate does not verify, Certificate error.
• A token refused later, because it expired or was revoked, fails the reading that met it with Authentication error. The next reading resolves the token again, and a refused token is sent again only once it resolves to another one.

The connection state follows the API server: the device is Connected once it reads who its token is, and whenever a reading succeeds. A server that refuses the account the cluster's version still connects, and the records then carry no observer.version.

Requests

• Lists are read whole, or not at all. Every dataset reads a list, a page of 500 objects at a time. A page that fails fails the reading, so a reading never takes part of a list for all of it, and no object is marked removed because a request failed. A continue token the server no longer holds (410) starts the list again, once.
• Deadlines. Every request has the device's timeout as its deadline, and stopping the device abandons the requests in flight.
• Only the configured server is called. The device builds every request itself on the address it was configured with, and never follows a redirect, so the token never leaves for another host. A page larger than 128 MiB is not read.
• User agent. The device identifies itself as VirtualMetric Director <version>.

When Things Are Read

• Every dataset on its own interval, all of them at once when the device starts. The hourly inventories do not hold back the events, which are read every minute.
• A failed reading is read again after 60 seconds when its interval is longer. A reading the server refused the account, or of an API the cluster does not serve, waits for its interval, since it would be refused again.
• What goes wrong in a dataset is its own. A refused list, records that cannot be written, or a panic costs that dataset's reading alone. The dataset keeps its checkpoint or its baseline, and the other datasets go on.

Tags

Every record carries, as searchable fields, the cluster it came through and where its object sits:

Field	On	Holds
observer.vendor, observer.product	Every record	Red Hat, OpenShift
observer.name	Every record	The device's name in the console: the cluster the record came through
observer.hostname or observer.ip	Every record	The device's address
observer.version	Every record	The cluster's version (4.14.8); left out when the account may not read it
openshift.project.id, .name	Pods, persistent volume claims, replication controllers, services, service accounts, events	The project the object is in
host.name	Nodes; pods on a node; events a kubelet reported	The node itself, the node a pod runs on, or the node that reported the event
openshift.node.id, .name	Pods on a node; events a kubelet reported	The node
cloud.availability_zone, cloud.region	Nodes; pods on a node; events a kubelet reported	The node's zone and region
openshift.workload.kind, .name	Pods	What runs the pod: the Deployment over its ReplicaSet, the DeploymentConfig over its replication controller, the CronJob over its Job, or else its owner itself. A static pod has none.

host.name is always a node, never a pod: a search for a node finds the node, the pods on it, and the events of its kubelet alike.

The projects, nodes, and workloads come from the device's discovery of the cluster, which it keeps in memory. The node dataset refreshes it at every reading, and any other dataset when it is an hour old. Nothing is registered from it.

event.dataset: "openshift.pod" and openshift.project.name: "web"
event.dataset: "openshift.event" and log.level: "warning"
host.name: "worker-0"
observer.name: "ocp01"

Records

Every record is one ECS 8.11 document:

• event.module is openshift, and event.dataset is openshift.<dataset>.
• event.kind is state for an inventory item, and event for an event.
• event.category is iam for users and service accounts, network for services and the network configuration, and host for the others.
• Fields ECS has no name for sit under openshift.<dataset>. Sizes and memory are bytes, CPU is cores (3.5), times are dates in UTC, and a field the cluster leaves empty is left out.

Inventory records carry event.action from a dataset's second reading on: added, changed, unchanged, or removed.

Each record also carries its dataset's own definition id. For the fields of each dataset, see Red Hat OpenShift Datasets.

Console

In the console, go to Fleet Management > Devices > Red Hat OpenShift and click Add new device. The wizard's steps are:

1. General settings, as on every device.
2. Connection settings: Address, Port, and Token; then TLS encryption, and while it is on, Skip certificate verification, CA certificate (Optional), Server name (Optional), Minimum TLS version (Optional), and Maximum TLS version (Optional). With TLS encryption off, the wizard warns that the token is sent in cleartext.
3. Advanced settings: Timeout and Initial lookback, both in seconds.
4. Data collection: the datasets or profile the device reads. A device receives datasets or a profile, never both.
5. Review and create.

The token is masked where it is typed and on the device page, and the review shows only whether it is set.

Simulation

The Red Hat OpenShift device is not simulated. What it writes is decided by its datasets, eleven record shapes under eleven definition ids, and one generic sample cannot stand in for them. A simulation block on a Red Hat OpenShift device is refused on save. See Simulation.

Examples

Service Account Token in a Vault

Reading the nodes, projects, pods, and events of one cluster with a service account's token kept in a vault...

devices: - id: 512300170 name: ocp01 type: openshift status: true properties: address: "api.ocp01.example.com" token: "$secret{store=corp-vault,ref=openshift/ocp01}" definitions: - name: openshift_node_collector - name: openshift_pod_collector - name: openshift_project_collector - name: openshift_event_collector

Pinned Cluster CA

Verifying the API server against the cluster's own CA, with the token in an environment variable, and starting the events an hour back...

devices: - id: 512300171 name: ocp02 type: openshift status: true properties: address: "api.ocp02.example.com" token: "${OPENSHIFT_TOKEN}" initial_lookback: 3600 tls: ca_name: "ocp02-ca.pem" definitions: - name: openshift_node_collector - name: openshift_service_collector - name: openshift_user_collector - name: openshift_event_collector