Kubernetes

Pull

Synopsis

Director polls one Kubernetes cluster through its API server and writes what it reads as ECS documents: the cluster's namespaces, nodes, and pods, its Deployments, DaemonSets, and StatefulSets, its services, ingresses, and service accounts, its role bindings and cluster role bindings, its persistent volume claims and CSI drivers, and its events. The cluster may be EKS, AKS, GKE, RKE, k3s, kubeadm, or any other. Nothing is installed in the cluster.

A cluster's nodes, pods, workloads, services, and ingresses are the inventory a security team asks for first: what runs where, what is exposed, and which pods share the host's network. Its role bindings say who may do what in it, and its events say what happened to all of them.

What the device reads is decided by the datasets and profiles assigned to it, as for Windows, Linux, and SNMP devices. Each data model is a dataset of its own — see Kubernetes Datasets.

The cluster is the device. One device reads every node and every namespace of one cluster, or the namespaces it names (see Namespaces and Label Selector).

Nothing it finds is registered as a device. The nodes, namespaces, and workloads the device discovers are records of the cluster's, and they tag the records of the objects inside them (see Tags). None of them becomes a device of its own.

For an OpenShift cluster, use the Red Hat OpenShift device. The two devices share one reader of what an OpenShift cluster and any other Kubernetes cluster have in common, and their records of the same models differ only in their module (kubernetes.*, openshift.*).

The device dials the API server and binds no port, so any number of Kubernetes devices may use port 6443 or 443. One Director polls each device: two Directors never read the same events twice.

note

The Kubernetes device needs Director 2.3.1 or later. With no Director or cluster at that version selected, the wizard shows Kubernetes devices are unavailable on directors or clusters below version 2.3.1 and the device cannot be created.

What the Linux Agent Reads Instead

The device has no metric dataset. Usage, and the pods' logs, are read on every node by the Linux agent, and the device does not read them again:

Node, pod, and container usage, and their readiness and restarts as metric series over time, come from the Linux agent's Kubernetes Metrics dataset. The Kubernetes node and pod inventories carry a node's readiness and a pod's phase and restarts too, but as fields of each object's record at every reading, not as metrics.
The pods' logs come from the Linux agent's Kubernetes Pod Logs dataset.

The Kubernetes device reads what those datasets do not: the cluster as a whole, object by object, from Director, and what happens in it.

Signing In

Every request sends the device's credentials. The device reads; it changes nothing, and never lists a secret.

A token — the device's token, or the kubeconfig user's — is sent as Authorization: Bearer <token>. It is resolved at every request, so a token rotated in a vault is sent once the vault's cache lets it through. The configuration keeps the reference, and a changed reference restarts the device.
A client certificate in the kubeconfig's user (client-certificate and client-key, or their -data forms) signs in over mutual TLS with no token, as a k3s, kubeadm, or RKE2 admin kubeconfig does. A client certificate needs https://.
A kubeconfig gives its context's cluster (server, certificate-authority or certificate-authority-data, insecure-skip-tls-verify, tls-server-name) and its user. It is read when the device starts; only its token is read again at every request, so a changed server or certificate takes a restart of the device. A path in it is read under Director's service root. A context's namespace is not read: the device's namespaces is.
The device's settings win over the kubeconfig's. url, token, and the tls keys take the place of what the kubeconfig says, as kubectl's flags do. With token set, the context's user is not read at all, as kubectl --token takes its place.
A proxy — the kubeconfig cluster's proxy-url, http://, https://, or socks5://, with the proxy's own user and password when its URL has them — carries every request, url's too. The API server's TLS runs end to end through it, so the proxy sees the API server's host and port, never the token. An API server called over http:// sends the token through the proxy in cleartext. An https:// proxy's certificate is verified with the API server's TLS settings: its CA, and the name tls-server-name gives. The Director's NO_PROXY does not apply to it. A proxy-url of another scheme, or with no host, is a Configuration error whose message does not name the URL, which may hold a password.

Not read: a user that signs in through a program (exec, as aws eks get-token, kubelogin, and gke-gcloud-auth-plugin do), an auth-provider, a tokenFile, a username and password, and impersonation (as, as-uid, as-groups, as-user-extra). A kubeconfig that asks for one of these is refused, naming it, unless the device has its own token, which takes the user's place. A cluster's proxy-url is read (above).

EKS, AKS, and GKE. Their kubeconfigs sign in through a program; see Managed Clusters: EKS, AKS, and GKE.

Account

A service account bound to the cluster role below by a ClusterRoleBinding reads every dataset. The device only lists, so every rule is list, except get on namespaces, which reads each namespace the namespaces property names:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: virtualmetric-reader
rules:
  - apiGroups: [""]
    resources: [nodes, pods, services, serviceaccounts, persistentvolumeclaims, events]
    verbs: [list]
  - apiGroups: [""]
    resources: [namespaces]
    verbs: [get, list]
  - apiGroups: [storage.k8s.io]
    resources: [csidrivers]
    verbs: [list]
  - apiGroups: [apps]
    resources: [replicasets, deployments, daemonsets, statefulsets]
    verbs: [list]
  - apiGroups: [batch]
    resources: [jobs]
    verbs: [list]
  - apiGroups: [networking.k8s.io]
    resources: [ingresses]
    verbs: [list]
  - apiGroups: [rbac.authorization.k8s.io]
    resources: [rolebindings, clusterrolebindings]
    verbs: [list]

It grants no secrets, ConfigMaps, pod logs, or exec, and no verb that writes. It must be a cluster role: the nodes, the CSI drivers, the cluster role bindings, and the list of every namespace belong to the cluster, not to a namespace, and only a cluster role bound cluster-wide can grant them.

ReplicaSets and Jobs are read only to tag a pod with the Deployment or CronJob that runs it (see Tags). Without them, the pods are written without their Deployment or CronJob.
A dataset left disabled needs no rule; leave its rule out to narrow the role further.

Save the role as virtualmetric-reader.yaml and apply it first, so it exists when the binding names it; then create the account, bind it, and give it a token:

kubectl apply -f virtualmetric-reader.yaml
kubectl create namespace virtualmetric
kubectl create serviceaccount virtualmetric -n virtualmetric
kubectl create clusterrolebinding virtualmetric-reader --clusterrole=virtualmetric-reader --serviceaccount=virtualmetric:virtualmetric
kubectl create token virtualmetric -n virtualmetric --duration=8760h

kubectl create token gives a token that expires, and the API server may cap its duration. For a long-lived token, see A Long-Lived Token.
The built-in view cluster role is not enough: it reads neither the nodes, the CSI drivers, nor either kind of binding, so four datasets would be refused. Bound cluster-wide, it also reads every ConfigMap and every pod's logs, which the device does not need.

An account limited to some namespaces: bind the same cluster role with a RoleBinding in each namespace, and name those namespaces in namespaces. The account reads what runs there, those namespaces, and their events, and is refused the nodes, the CSI drivers, and the cluster role bindings.

A dataset the account may not read keeps the device Connected. The device's log says so once, and its message names each such dataset, and why, until that dataset reads again:

Kubernetes poller connected to eks-prod. kubernetes_node_collector reads nothing: the API server refused the account /api/v1/nodes (403 Forbidden)

A Long-Lived Token

kubectl create token asks the API server for a token that expires, and the server may cap how long it lives. A device that reads for months takes a long-lived token: a secret of type kubernetes.io/service-account-token that names the account, into which the cluster writes the account's token and its own CA, ca.crt, a moment after it is made. The token lives until the secret, or the account, is deleted, which revokes it: the API server refuses it some seconds later. The cleaner that retires unused legacy tokens (Kubernetes 1.30) retires only those the cluster made of itself for an account, never one made like this.

Save the secret as virtualmetric-token.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: virtualmetric-token
  namespace: virtualmetric
  annotations:
    kubernetes.io/service-account.name: virtualmetric
type: kubernetes.io/service-account-token

Apply it, then, once the cluster has filled it in, read its token and ca.crt:

kubectl apply -f virtualmetric-token.yaml
kubectl -n virtualmetric get secret virtualmetric-token -o jsonpath='{.data.token}' | base64 -d > token
kubectl -n virtualmetric get secret virtualmetric-token -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

The token is the device's token, best kept in a vault ($secret{...}), and ca.crt the CA the API server's certificate is verified against (tls.ca_name), unless that certificate is a public CA's, as on GKE's DNS-based endpoint (see Managed Clusters: EKS, AKS, and GKE).

Managed Clusters: EKS, AKS, and GKE

A managed cluster's own kubeconfig signs in by running a program (exec), which the device never runs. The device reads a managed cluster as it reads any other: as a service account bound to virtualmetric-reader, with a long-lived token. The cluster's admin makes them once, with kubectl signed in the cloud's own way, and the device needs nothing of the cloud's identity system:

Sign kubectl in to the cluster as an admin who may make cluster roles and their bindings (per cloud, below), and make what Account and A Long-Lived Token show: the virtualmetric namespace and service account, the role and its binding, and the token secret.
Give the device the API server's URL (per cloud, below), the token as token, and ca.crt as tls.ca_name. Or give it the cloud's own kubeconfig as kubeconfig, for its server and its CA, and the token as token, which takes the place of the kubeconfig's exec user. Or make one kubeconfig of the three, for one vault entry. GKE's DNS-based endpoint is the exception: a public CA signs its certificate, so pin no CA there — no tls.ca_name, and no --certificate-authority below:
```
kubectl config --kubeconfig=virtualmetric.kubeconfig set-cluster prod --server=<the API server's URL> --certificate-authority=ca.crt --embed-certs=true
kubectl config --kubeconfig=virtualmetric.kubeconfig set-credentials virtualmetric --token="$(cat token)"
kubectl config --kubeconfig=virtualmetric.kubeconfig set-context prod --cluster=prod --user=virtualmetric
kubectl config --kubeconfig=virtualmetric.kubeconfig use-context prod
```
Let the Director reach the API server: each cloud admits only the networks it is told to.

Cloud	The API server's URL, and its CA	The admin who runs `kubectl`	The device's account	The Director's network
EKS	`aws eks describe-cluster --name prod --query cluster.endpoint`; its CA is `cluster.certificateAuthority.data`, the same as the token secret's `ca.crt`	An IAM principal whose access entry grants cluster admin (the cluster's creator, unless it was made without one, or one with the `AmazonEKSClusterAdminPolicy` access policy), or a `system:masters` line in the `aws-auth` ConfigMap	Needs no access entry and no `aws-auth` line: those map IAM principals, and a service account is the cluster's own identity, whose token the API server checks itself	The public endpoint, with the Director's address among its public access CIDRs, or the private endpoint, from the cluster's VPC or a network joined to it
AKS	`https://` and `az aks show -g <group> -n prod --query fqdn` (`privateFqdn` for a private cluster); its CA is the one `az aks get-credentials` writes, the same as the token secret's `ca.crt`	A user `kubectl` signs in with `az aks get-credentials` and `kubelogin` who may make cluster roles (Azure Kubernetes Service RBAC Cluster Admin where Microsoft Entra ID authorization is on, else a `cluster-admin` binding), or the admin credentials (`--admin`) where local accounts are on	Kubernetes RBAC authorizes it, beside Microsoft Entra ID authorization, which governs Entra identities. Turning local accounts off (`--disable-local-accounts`) takes away the cluster's static admin and user credentials, not its service accounts' tokens	The Director's address among the cluster's authorized IP ranges, or a private cluster's network
GKE	The IP endpoint, `https://` and `gcloud container clusters describe prod --location <location> --format='value(endpoint)'`, whose CA is `masterAuth.clusterCaCertificate`, the same as the token secret's `ca.crt`; or the DNS-based endpoint, `controlPlaneEndpointsConfig.dnsEndpointConfig.endpoint` (`gke-<id>.<location>.gke.goog`), whose certificate a public Google CA signs, so no CA is pinned	A user `kubectl` signs in with `gcloud container clusters get-credentials` and `gke-gcloud-auth-plugin`, with the Kubernetes Engine Admin role or a `cluster-admin` binding	Kubernetes RBAC authorizes it. The IP endpoint takes its token; the DNS-based endpoint takes a service account's token only once the cluster lets it (`gcloud container clusters update prod --enable-dns-access --enable-k8s-tokens-via-dns`)	The IP endpoint, with the Director's address among the cluster's authorized networks; or the DNS-based endpoint

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: kubernetes
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    token: <string>
    kubeconfig: <string>
    context: <string>
    namespaces: <string[]>
    label_selector: <string>
    timeout: <numeric>
    initial_lookback: <numeric>
    tls:
      insecure_skip_verify: <boolean>
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `kubernetes`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Connection

Field	Required	Default	Description
`url`	N	the kubeconfig's server	The API server, `https://<host>[:<port>][/<path>]`. A URL with no scheme is `https://`; `http://` calls the API over plain HTTP. A path is where a proxy serves the API, such as Rancher's `/k8s/clusters/<cluster id>`. A wildcard host, a user, a query, or a fragment is refused. Required unless a `kubeconfig` names the server.
`token`	N	-	A service account's bearer token. Required unless a `kubeconfig` is given; beside one, it takes the place of the kubeconfig's user. See Signing In.
`kubeconfig`	N	-	A kubeconfig: its YAML or JSON, a path under the service root, `${ENV_VAR}`, or `$secret{...}`. A `token` or a `kubeconfig` is required.
`context`	N	the kubeconfig's `current-context`	The kubeconfig's context the device reads: its cluster, and its user unless `token` is set

The token takes plain text, ${ENV_VAR}, or $secret{...} ($secret{id=...} or $secret{store=...,ref=...}).

TLS

Whether the API is called over TLS is the URL's scheme; there is no tls.status.

Field	Required	Default	Description
`tls.insecure_skip_verify`	N	the kubeconfig's `insecure-skip-tls-verify`, else `false`	`true` accepts any certificate. Pin the CA instead where you can. Given either way, it wins over the kubeconfig's.
`tls.ca_name`	N	the kubeconfig's CA, else the system trust store	A CA to verify the API server's certificate against
`tls.server_name`	N	the kubeconfig's `tls-server-name`, else the URL's host	The name the certificate is verified for, when it is not the URL's host
`tls.min_tls_version`	N	`tls1.2`	`tls1.0`, `tls1.1`, `tls1.2`, or `tls1.3`
`tls.max_tls_version`	N	-	No cap

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Pin the cluster's CA. An API server's certificate is signed by the cluster's own CA, a managed cluster's included, unless it was given one from a public CA. Set tls.ca_name to the certificate-authority-data of the cluster's kubeconfig, decoded from base64, or a token secret's ca.crt (see A Long-Lived Token), or give the kubeconfig, rather than turning verification off. GKE's DNS-based endpoint has a public CA's certificate: pin none.

warning

Over an http:// URL, the bearer token is sent in cleartext with every request, and anyone who reads it can call the API server as the account until the token expires or is revoked. The device logs a warning once when it starts, unless the URL's host is the Director's own or a loopback address. Use plain HTTP only where the network between the Director and the proxy that terminates TLS is trusted.

Datasets

The device's definitions are its datasets, each named after its type, kubernetes_<dataset>_collector. In the console they are the datasets and profiles assigned to the device. For the dataset types, their intervals, and their fields, see Kubernetes Datasets.

Advanced Configuration

Field	Required	Default	Description
`namespaces`	N	every namespace	The namespaces whose objects the device reads: a list, or names separated by commas or spaces. See Namespaces and Label Selector.
`label_selector`	N	-	Narrows what runs in the namespaces to the objects it selects, in the API's own syntax (`app.kubernetes.io/part-of=shop,tier notin (cache)`). A selector that does not parse is a configuration error.
`timeout`	N	`60`	Seconds each request may take, from `1` to `300`. A list is read a page of 500 objects at a time, each page its own request.
`initial_lookback`	N	`86400`	Seconds before its first reading where the event dataset starts when it has no checkpoint. `0` starts at the first reading. The dataset's own `initial_lookback` wins. The API server keeps an event an hour after it last happened by default, so a day reads every event it still holds.

Namespaces and Label Selector

namespaces names the namespaces whose objects the device reads. Each namespaced dataset — pods, workloads, services, ingresses, service accounts, role bindings, persistent volume claims, and events — lists each named namespace on its own, never every namespace's, so an account with a role in each of those namespaces alone is enough. The namespace dataset reads each named namespace's object. Without namespaces, every list is the cluster's.
label_selector narrows the namespaced objects — pods, workloads, services, ingresses, service accounts, role bindings, and persistent volume claims — as kubectl get -l does; the API server does the selecting. Nodes, namespaces, CSI drivers, cluster role bindings, and events are read whole.
A reading is read whole or not at all, across the namespaces too. A namespace whose list fails, or that the account may not read, fails the dataset's reading, so no object of another namespace is marked removed because one list failed.

Connection States

A configuration error — no URL and no kubeconfig, no credentials, a kubeconfig that cannot be read or signs in a way the device does not, a reference that does not resolve, a CA that cannot be read — stops the device with Configuration error. The message names the property but never its value.
At start the device reads the API server's version, as kubectl version does. Credentials the server refuses (401) set Authentication error and stop the device; the Director starts it again later.
A server that does not answer, or a gateway in front of it answering 502, 503, or 504, sets Connection error; one whose certificate does not verify, Certificate error.

The connection state follows the API server: the device is Connected once it reads the server's version, and whenever a reading succeeds. A server that refuses the account its version still connects, and the records then carry no observer.version.

Requests

Lists are read whole, or not at all. Every dataset reads a list, a page of 500 objects at a time. A page that fails fails the reading, so a reading never takes part of a list for all of it, and no object is marked removed because a request failed. A continue token the server no longer holds (410) starts the list again, once.
Deadlines. Every request has the device's timeout as its deadline, and stopping the device abandons the requests in flight.
Only the configured server is called. The device builds every request itself on the server it was configured with, and never follows a redirect, so the token never leaves for another host. A page larger than 128 MiB is not read.
User agent. The device identifies itself as VirtualMetric Director <version>.

When Things Are Read

Every dataset on its own interval, all of them at once when the device starts. The hourly inventories do not hold back the events, which are read every minute.
A failed reading is read again after 60 seconds when its interval is longer. A reading the server refused the account, or of an API the cluster does not serve, waits for its interval, since it would be refused again.
What goes wrong in a dataset is its own. A refused list, records that cannot be written, or a panic costs that dataset's reading alone. The dataset keeps its checkpoint or its baseline, and the other datasets go on.

Field	On	Holds
`observer.vendor`, `observer.product`	Every record	`CNCF`, `Kubernetes`. The distribution is in the version.
`observer.name`	Every record	The device's name in the console: the cluster the record came through
`observer.hostname` or `observer.ip`	Every record	The API server's host
`observer.version`	Every record	The API server's version (`v1.30.4-eks-a737599`, `v1.37.1+k3s1`); left out when the account may not read it
`kubernetes.namespace.id`, `.name`	Pods, workloads, persistent volume claims, services, ingresses, service accounts, role bindings, events	The namespace the object is in
`host.name`	Nodes; pods on a node; events a kubelet reported	The node itself, the node a pod runs on, or the node that reported the event
`kubernetes.node.id`, `.name`	Pods on a node; events a kubelet reported	The node
`cloud.availability_zone`, `cloud.region`	Nodes; pods on a node; events a kubelet reported	The node's zone and region
`kubernetes.workload.kind`, `.name`	Pods	What runs the pod: the Deployment over its ReplicaSet, the CronJob over its Job, or else its owner itself. A static pod has none.

Records

Every record is one ECS 8.11 document:

event.module is kubernetes, and event.dataset is kubernetes.<dataset>.
event.kind is state for an inventory item, and event for an event.
event.category is iam for service accounts and the bindings, network for services and ingresses, and host for the others.
Fields ECS has no name for sit under kubernetes.<dataset>. Sizes and memory are bytes, CPU is cores (3.5), times are dates in UTC, and a field the cluster leaves empty is left out.

Inventory records carry event.action from a dataset's second reading on: added, changed, unchanged, or removed.

Each record also carries its dataset's own definition id. For the fields of each dataset, see Kubernetes Datasets.

Console

In the console, go to Fleet Management > Devices > Kubernetes, listed beside Red Hat OpenShift, and click Add new device. The wizard's steps are:

General settings, as on every device.
Connection settings: URL, its scheme (HTTPS or HTTP) selected beside the address; Token, Kubeconfig (Optional), and Context (Optional); then, for HTTPS, the TLS settings: Skip certificate verification, the CA certificate, Server name (Optional), and the minimum and maximum TLS versions. With HTTP selected, the wizard warns that the token is sent in cleartext.
Advanced settings: Namespaces (Optional), Label selector (Optional), Timeout, and Initial lookback, the last two in seconds.
Data collection: the datasets or profile the device reads. A device receives datasets or a profile, never both.
Review and create.

The token and the kubeconfig are masked where they are typed and on the device page, and the review shows only whether each is set.

Simulation

The Kubernetes device is not simulated. What it writes is decided by its datasets, fourteen record shapes under fourteen definition ids, and one generic sample cannot stand in for them. See Simulation.

Limitations

The pods' logs and the nodes' and pods' usage are not read here; they are the Linux agent's Kubernetes datasets.
A kubeconfig that signs in through a program on its own is not read. Give the device a service account's token beside it.
Tested on k3s, kind, and Rancher. The device was run against k3s, kind (kubeadm), and a cluster behind a Rancher proxy. EKS, AKS, and GKE were not tried; their recipe follows each cloud's documentation. RKE1 is end of life; RKE2 runs the upstream API server.

Examples

Service Account Token in a Vault

Reading the nodes, namespaces, pods, and events of an EKS cluster with a service account's token kept in a vault, against the cluster's pinned CA...

devices:
  - id: 512300180
    name: eks-prod
    type: kubernetes
    status: true
    properties:
      url: "https://ABCDEF0123456789.gr7.eu-west-1.eks.amazonaws.com"
      token: "$secret{store=corp-vault,ref=kubernetes/eks-prod}"
      tls:
        ca_name: "eks-prod-ca"
    definitions:
      - name: kubernetes_node_collector
      - name: kubernetes_pod_collector
      - name: kubernetes_namespace_collector
      - name: kubernetes_event_collector

Kubeconfig, Two Namespaces

Reading two namespaces of an RKE cluster through a Rancher-made kubeconfig kept in a vault, narrowed to one application's objects...

devices:
  - id: 512300181
    name: rke-shop
    type: kubernetes
    status: true
    properties:
      kubeconfig: "$secret{store=corp-vault,ref=kubernetes/rke-shop-kubeconfig}"
      context: shop-prod
      namespaces: [shop, payments]
      label_selector: "app.kubernetes.io/part-of=shop"
    definitions:
      - name: kubernetes_namespace_collector
      - name: kubernetes_pod_collector
      - name: kubernetes_service_collector
      - name: kubernetes_event_collector

Synopsis​

What the Linux Agent Reads Instead​

Signing In​

Account​

A Long-Lived Token​

Managed Clusters: EKS, AKS, and GKE​

Schema​

Configuration​

Device​

Connection​

TLS​

Datasets​

Advanced Configuration​

Namespaces and Label Selector​

Connection States​

Requests​

When Things Are Read​

Tags​

Records​

Console​

Simulation​

Limitations​

Examples​

Service Account Token in a Vault​

Kubeconfig, Two Namespaces​