Instrumentation

• 1: Kubernetes Component SLI Metrics
• 2: CRI Pod & Container Metrics
• 3: Native Histogram Support for Kubernetes Metrics
• 4: Node metrics data
• 5: Understand Pressure Stall Information (PSI) Metrics
• 6: Kubernetes z-pages
• 7: Kubernetes Metrics Reference

1 - Kubernetes Component SLI Metrics

By default, Kubernetes 1.36 publishes Service Level Indicator (SLI) metrics for each Kubernetes component binary. This metric endpoint is exposed on the serving HTTPS port of each component, at the path /metrics/slis. The ComponentSLIs feature gate defaults to enabled for each Kubernetes component as of v1.27.

SLI Metrics

With SLI metrics enabled, each Kubernetes component exposes two metrics, labeled per healthcheck:

• a gauge (which represents the current state of the healthcheck)
• a counter (which records the cumulative counts observed for each healthcheck state)

You can use the metric information to calculate per-component availability statistics. For example, the API server checks the health of etcd. You can work out and report how available or unavailable etcd has been - as reported by its client, the API server.

The prometheus gauge data looks like this:

# HELP kubernetes_healthcheck [ALPHA] This metric records the result of a single healthcheck.
# TYPE kubernetes_healthcheck gauge
kubernetes_healthcheck{name="autoregister-completion",type="healthz"} 1
kubernetes_healthcheck{name="autoregister-completion",type="readyz"} 1
kubernetes_healthcheck{name="etcd",type="healthz"} 1
kubernetes_healthcheck{name="etcd",type="readyz"} 1
kubernetes_healthcheck{name="etcd-readiness",type="readyz"} 1
kubernetes_healthcheck{name="informer-sync",type="readyz"} 1
kubernetes_healthcheck{name="log",type="healthz"} 1
kubernetes_healthcheck{name="log",type="readyz"} 1
kubernetes_healthcheck{name="ping",type="healthz"} 1
kubernetes_healthcheck{name="ping",type="readyz"} 1

While the counter data looks like this:

# HELP kubernetes_healthchecks_total [ALPHA] This metric records the results of all healthcheck.
# TYPE kubernetes_healthchecks_total counter
kubernetes_healthchecks_total{name="autoregister-completion",status="error",type="readyz"} 1
kubernetes_healthchecks_total{name="autoregister-completion",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="autoregister-completion",status="success",type="readyz"} 14
kubernetes_healthchecks_total{name="etcd",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="etcd",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="etcd-readiness",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="informer-sync",status="error",type="readyz"} 1
kubernetes_healthchecks_total{name="informer-sync",status="success",type="readyz"} 14
kubernetes_healthchecks_total{name="log",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="log",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="ping",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="ping",status="success",type="readyz"} 15

Using this data

The component SLIs metrics endpoint is intended to be scraped at a high frequency. Scraping at a high frequency means that you end up with greater granularity of the gauge's signal, which can be then used to calculate SLOs. The /metrics/slis endpoint provides the raw data necessary to calculate an availability SLO for the respective Kubernetes component.

2 - CRI Pod & Container Metrics

The kubelet collects pod and container metrics via cAdvisor. As an alpha feature, Kubernetes lets you configure the collection of pod and container metrics via the Container Runtime Interface (CRI). You must enable the PodAndContainerStatsFromCRI feature gate and use a compatible CRI implementation (containerd >= 1.6.0, CRI-O >= 1.23.0) to use the CRI based collection mechanism.

CRI Pod & Container Metrics

With PodAndContainerStatsFromCRI enabled, the kubelet polls the underlying container runtime for pod and container stats instead of inspecting the host system directly using cAdvisor. The benefits of relying on the container runtime for this information as opposed to direct collection with cAdvisor include:

• Potential improved performance if the container runtime already collects this information during normal operations. In this case, the data can be re-used instead of being aggregated again by the kubelet.

• It further decouples the kubelet and the container runtime allowing collection of metrics for container runtimes that don't run processes directly on the host with kubelet where they are observable by cAdvisor (for example: container runtimes that use virtualization).

3 - Native Histogram Support for Kubernetes Metrics

Kubernetes components can expose histogram metrics in Prometheus Native Histogram format, alongside the classic histogram format. Native histograms use exponential bucket boundaries instead of fixed boundaries, providing significant storage efficiency, improved query performance, and finer-grained visibility into distributions.

Before you begin

To use native histograms, you need:

• Kubernetes v1.36 or later with the NativeHistograms feature gate enabled.
• Prometheus 2.40 or later to scrape and store native histograms. Prometheus 3.0+ is recommended for per-job configuration.

What are native histograms?

Classic Prometheus histograms use fixed bucket boundaries (for example, [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds). Each bucket creates a separate time series (_bucket, _count, _sum), which can lead to:

• High storage costs at scale, because each histogram generates many time series.
• Accuracy issues, because data points within a wide bucket range are indistinguishable. For example, a request completing in 1µs and one completing in 4ms both fall into the same le="0.005" bucket.

Native histograms address these limitations by using exponential bucket boundaries that automatically adjust to the data distribution. Benefits include:

• ~10x reduction in time series count per histogram metric, significantly reducing Prometheus storage and improving query performance.
• Finer-grained resolution for detecting performance regressions and setting precise SLO thresholds.

How it works

When the NativeHistograms feature gate is enabled, Kubernetes components expose histogram metrics in both classic and native formats simultaneously (dual exposition). The format returned depends on the Accept header in the HTTP request (Prometheus content negotiation). Prometheus sets this header automatically based on your scrape configuration; you only need to be aware of it when querying the /metrics endpoint directly.

• Text format (Accept: text/plain, OpenMetrics 1.0): Returns only classic histogram buckets. Backward compatible with all existing tooling.

  # Classic histogram buckets (always present)
  apiserver_request_duration_seconds_bucket{le="0.005"} 1000
  apiserver_request_duration_seconds_bucket{le="0.01"} 2000
  ...
  apiserver_request_duration_seconds_bucket{le="+Inf"} 10000
  apiserver_request_duration_seconds_count 10000
  apiserver_request_duration_seconds_sum 450.5
  

• Protobuf format (Accept: application/vnd.google.protobuf): Contains both classic buckets and native histogram data. Prometheus automatically requests this format when scrape_native_histograms: true is set in the Prometheus scrape configuration for the corresponding scrape job.

This dual exposition strategy ensures:

• Existing dashboards and alerts continue to work without changes.
• Users can migrate queries to native histograms at their own pace.
• Prometheus stores whichever format it is configured to ingest.

Enabling native histograms

Enabling native histograms is a two-step process: enable the feature gate on Kubernetes components, and configure Prometheus to scrape native histograms.

Step 1: Enable the Kubernetes feature gate

Enable the NativeHistograms feature gate on the Kubernetes components you want to expose native histograms from:

--feature-gates=NativeHistograms=true

This feature gate applies to the following components:

• kube-apiserver
• kube-controller-manager
• kube-scheduler
• kubelet
• kube-proxy

Each component's metrics are independent; you can enable or disable the feature gate per component.

Step 2: Configure Prometheus

The Prometheus configuration depends on your Prometheus version.

For Prometheus 3.x, use per-job configuration for fine-grained control:

scrape_configs:
  - job_name: 'kubernetes-apiservers'
    scrape_native_histograms: true            # Ingest native histograms
    always_scrape_classic_histograms: true     # Keep classic format during migration

Set both options to true during the migration period. This allows you to ingest native histograms while keeping classic histograms for existing dashboards.

Migrating dashboards and alerts

When migrating from classic to native histogram queries, follow this workflow:

1. Enable both formats: Set scrape_native_histograms: true and always_scrape_classic_histograms: true in your Prometheus scrape config.

2. Migrate queries: Update dashboard queries and alert expressions from classic histogram functions to native histogram equivalents.

   Classic query:

   histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
   

   Native histogram query:

   histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))
   

3. Verify in staging: Test all dashboards and alerts with native histogram queries before rolling out to production.

4. Disable classic scraping: Once migration is complete and verified, set always_scrape_classic_histograms: false to reduce storage overhead.

Disabling native histograms

You can disable native histograms at any time using either of two approaches:

• Prometheus-side (fastest, no Kubernetes restart needed; Prometheus 3.x only): Set scrape_native_histograms: false per scrape job. Prometheus resumes scraping classic format on the next scrape interval.

• Kubernetes feature gate: Restart the component with --feature-gates=NativeHistograms=false. Only classic histogram format is exposed after restart.

When native histograms are disabled, the metrics endpoint reverts to classic histogram format only. Historical native histogram data in Prometheus remains queryable.

Troubleshooting

• Dashboards show no data after enabling native histograms : This occurs when Prometheus is configured with scrape_native_histograms: true but always_scrape_classic_histograms: false (the default), and your dashboards still use classic histogram queries (for example, histogram_quantile(..._bucket...)).

  Fix: Set always_scrape_classic_histograms: true to restore classic format ingestion while you migrate dashboards.

• Memory usage increase after enabling native histograms : A small memory increase is expected for native histogram bucket storage, bounded by a maximum of 160 buckets per histogram. Monitor process_resident_memory_bytes for unexpected increases.

  Fix: If memory pressure is severe, disable native histogram ingestion in Prometheus (scrape_native_histograms: false) or disable the Kubernetes feature gate.

• Prometheus logs errors about unknown metric format : Your Prometheus version is too old to understand native histograms.

  Fix: Upgrade Prometheus to 2.40+ or disable native histograms in Kubernetes.

• Not sure if native histograms are being exposed : Check the feature gate status by querying kubernetes_feature_enabled{name="NativeHistograms"} in Prometheus. A value of 1 indicates the feature is enabled. You can also query the metrics endpoint directly with protobuf format:

  curl -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
    https://<component-address>/metrics
  

  The response should contain native histogram encoding for histogram metrics.

References

• Read the Prometheus Native Histograms documentation for details on the native histogram format and query functions.
• See the Kubernetes metrics reference for the full list of metrics exposed by Kubernetes components.

4 - Node metrics data

The kubelet gathers metric statistics at the node, volume, pod and container level, and emits this information in the Summary API.

You can send a proxied request to the stats summary API via the Kubernetes API server.

Here is an example of a Summary API request for a node named minikube:

kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary"

Here is the same API call using curl:

# You need to run "kubectl proxy" first
# Change 8080 to the port that "kubectl proxy" assigns
curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary

Summary metrics API source

By default, Kubernetes fetches node summary metrics data using an embedded cAdvisor that runs within the kubelet. If you enable the PodAndContainerStatsFromCRI feature gate in your cluster, and you use a container runtime that supports statistics access via Container Runtime Interface (CRI), then the kubelet fetches Pod- and container-level metric data using CRI, and not via cAdvisor.

Pressure Stall Information (PSI)

As a stable feature, Kubernetes lets you configure kubelet to collect Linux kernel Pressure Stall Information (PSI) for CPU, memory, and I/O usage. The information is collected at node, pod and container level. See Summary API for detailed schema. Starting with Kubernetes v.1.36, the KubeletPSI feature gate is locked to true and cannot be disabled. The information is also exposed in Prometheus metrics.

You can learn how to interpret the PSI metrics in Understand PSI Metrics.

Requirements

Pressure Stall Information requires:

• Linux kernel versions 4.20 or later.
• cgroup v2

What's next

The task pages for Troubleshooting Clusters discuss how to use a metrics pipeline that rely on these data.

5 - Understand Pressure Stall Information (PSI) Metrics

As a stable feature, Kubernetes lets you configure the kubelet to collect Linux kernel Pressure Stall Information (PSI) for CPU, memory, and I/O usage. The information is collected at node, pod and container level. Starting with Kubernetes v.1.36, the KubeletPSI feature gate is locked to true and cannot be disabled.

PSI metrics are exposed through two different sources:

• The kubelet's Summary API, which provides PSI data at the node, pod, and container level.
• The /metrics/cadvisor endpoint on the kubelet, which exposes PSI metrics in the Prometheus format.

Requirements

Pressure Stall Information requires the following on your Linux nodes:

• The Linux kernel must be version 4.20 or newer.
• The kernel must be compiled with the CONFIG_PSI=y option. Most modern distributions enable this by default. You can check your kernel's configuration by running zgrep CONFIG_PSI /proc/config.gz.
• Some Linux distributions may compile PSI into the kernel but disable it by default. If so, you need to enable it at boot time by adding the psi=1 parameter to the kernel command line.
• The node must be using cgroup v2.

Understanding PSI Metrics

Pressure Stall Information (PSI) metrics are provided for three resources: CPU, memory, and I/O. They are categorized into two main types of pressure: some and full.

• some: This value indicates that some tasks (one or more) are stalled on a resource. For example, if some tasks are waiting for I/O, this metric will increase. This can be an early indicator of resource contention.
• full: This value indicates that all non-idle tasks are stalled on a resource simultaneously. This signifies a more severe resource shortage, where the entire system is unable to make progress.

Each pressure type provides four metrics: avg10, avg60, avg300, and total. The avg values represent the percentage of wall-clock time that tasks were stalled over 10-second, 60-second, and 5-minute moving averages. The total value is a cumulative counter in microseconds showing the total time tasks have been stalled.

Let's take for example the following query from the Summary API:
kubectl get --raw "/api/v1/nodes/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')/proxy/stats/summary" | jq '.pods[].containers[] | select(.name=="<CONTAINER_NAME>") | {name, cpu: .cpu.psi, memory: .memory.psi, io: .io.psi}'. This returns the information in a json format as such.

{
  "name": "<CONTAINER_NAME>",
  "cpu": {
    "full": {
      "total": 0,
      "avg10": 0,
      "avg60": 0,
      "avg300": 0
    },
    "some": {
      "total": 35232438,
      "avg10": 0.74,
      "avg60": 0.52,
      "avg300": 0.21,
    },  
  },
  "memory": {
    "full": {
      "total": 539105,
      "avg10": 0,
      "avg60": 0,
      "avg300": 0
    },
    "some": {
      "total": 658164,
      "avg10": 0.01,
      "avg60": 0.01,
      "avg300": 0.00,
    },
    }
  },
  "io": {
    "full": {
      "total": 33190987,
      "avg10": 0.31,
      "avg60": 0.22,
      "avg300": 0.05,
    },
    "some": {
      "total": 40809937,
      "avg10": 0.52,
      "avg60": 0.45,
      "avg300": 0.12,
    }
  }
}

Here is a simple spike scenario. The cpu.some avg10 value of 0.74 indicates that in the last 10 seconds, at least one task in this container was stalled on the CPU for 0.74% of the time (0.0074 seconds or 74 milliseconds). Because avg10 (0.74) is significantly higher than avg300 (0.21) on the same resource, this suggests a recent surge in resource contention rather than a sustained long-term bottleneck. If monitored continuously and the avg300 metrics increase as well, we can diagnose a more serious, lasting issue!

Additionally, notice how in this example cpu.some shows pressure, while cpu.full remains at 0.00. This tells us that while some processes were delayed waiting for CPU time, the container as a whole was still making progress. A non-zero full value would indicate that all non-idle tasks were stalled simultaneously, a much bigger problem. Although not as human-readable, the total value of 35232438 represents the cumulative stall time in microseconds, that allow latency spike detection that otherwise may not show in the averages.

As a final note, when observing high I/O Pressure alongside low Memory Pressure, it can indicate that the application is waiting on disk throughput rather than failing due to a lack of available RAM. The node is not over-committed on memory, and a different diagnosis for disk consumption can be investigated.

Example Scenarios

You can use a simple Pod with a stress-testing tool to generate resource pressure and observe the PSI metrics. The following examples use the agnhost container image, which includes the stress tool.

Generating CPU Pressure

Create a Pod that generates CPU pressure using the stress utility. This workload will put a heavy load on one CPU core.

Create a file named cpu-pressure-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: cpu-pressure-pod
spec:
  restartPolicy: Never
  containers:
  - name: cpu-stress
    image: registry.k8s.io/e2e-test-images/agnhost:2.47
    args:
    - "stress"
    - "--cpus"
    - "1"
    resources:
      limits:
        cpu: "500m"
      requests:
        cpu: "500m"

Apply it to your cluster: kubectl apply -f cpu-pressure-pod.yaml

Observing CPU Pressure

After the Pod is running, you can observe the CPU pressure through either the Summary API or the Prometheus metrics endpoint.

Using the Summary API:

Watch the summary stats for your node. In a separate terminal, run:

# Replace <node-name> with the name of a node in your cluster
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("cpu-pressure-pod"))'

You will see the some PSI metrics for CPU increase in the summary API output. The avg10 value for some pressure should rise above zero, indicating that tasks are spending time stalled on the CPU.

Using the Prometheus metrics endpoint:

Query the /metrics/cadvisor endpoint to see the container_pressure_cpu_waiting_seconds_total metric.

# Replace <node-name> with the name of the node where the pod is running
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \
    grep 'container_pressure_cpu_waiting_seconds_total{container="cpu-stress"'

The output should show an increasing value, indicating that the container is spending time stalled waiting for CPU resources.

Cleanup

Clean up the Pod when you are finished:

kubectl delete pod cpu-pressure-pod

Generating Memory Pressure

This example creates a Pod that continuously writes to files in the container's writable layer, causing the kernel's page cache to grow and forcing memory reclamation, which generates pressure.

Create a file named memory-pressure-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: memory-pressure-pod
spec:
  restartPolicy: Never
  containers:
  - name: memory-stress
    image: registry.k8s.io/e2e-test-images/agnhost:2.47
    command: ["/bin/sh", "-c"]
    args:
    - "i=0; while true; do dd if=/dev/zero of=testfile.$i bs=1M count=50 &>/dev/null; i=$(((i+1)%5)); sleep 0.1; done"
    resources:
      limits:
        memory: "200M"
      requests:
        memory: "200M"

Apply it to your cluster: kubectl apply -f memory-pressure-pod.yaml

Observing Memory Pressure

Using the Summary API:

In the summary output, you will observe an increase in the full PSI metrics for memory, indicating that the system is under significant memory pressure.

# Replace <node-name> with the name of a node in your cluster
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("memory-pressure-pod"))'

Using the Prometheus metrics endpoint:

Query the /metrics/cadvisor endpoint to see the container_pressure_memory_waiting_seconds_total metric.

# Replace <node-name> with the name of the node where the pod is running
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \
    grep 'container_pressure_memory_waiting_seconds_total{container="memory-stress"'

In the output, you will observe an increasing value for the metric, indicating that the system is under significant memory pressure.

Cleanup

Clean up the Pod when you are finished:

kubectl delete pod memory-pressure-pod

Generating I/O Pressure

This Pod generates I/O pressure by repeatedly writing a file to disk and using sync to flush the data from memory, which creates I/O stalls.

Create a file named io-pressure-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: io-pressure-pod
spec:
  restartPolicy: Never
  containers:
  - name: io-stress
    image: registry.k8s.io/e2e-test-images/agnhost:2.47
    command: ["/bin/sh", "-c"]
    args:
      - "while true; do dd if=/dev/zero of=testfile bs=1M count=128 &>/dev/null; sync; rm testfile &>/dev/null; done"

Apply this to your cluster: kubectl apply -f io-pressure-pod.yaml

Observing I/O Pressure

Using the Summary API:

You will see the some PSI metrics for I/O increase as the Pod continuously writes to disk.

# Replace <node-name> with the name of a node in your cluster
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/stats/summary" | jq '.pods[] | select(.podRef.name | contains("io-pressure-pod"))'

Using the Prometheus metrics endpoint:

Query the /metrics/cadvisor endpoint to see the container_pressure_io_waiting_seconds_total metric.

# Replace <node-name> with the name of the node where the pod is running
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/metrics/cadvisor" | \
    grep 'container_pressure_io_waiting_seconds_total{container="io-stress"'

You will see the metric's value increase as the Pod continuously writes to disk.

Cleanup

Clean up the Pod when you are finished:

kubectl delete pod io-pressure-pod

What's next

The task pages for Troubleshooting Clusters discuss how to use a metrics pipeline that rely on these data.

6 - Kubernetes z-pages

Kubernetes core components can expose a suite of z-endpoints to make it easier for users to debug their cluster and its components. These endpoints are strictly to be used for human inspection to gain real time debugging information of a component binary. In Kubernetes 1.36 these are beta features.

z-pages

Kubernetes v1.36 allows you to enable z-pages to help you troubleshoot problems with its core control plane components. These special debugging endpoints provide internal information about running components. For Kubernetes 1.36, components serve the following endpoints (when enabled):

• z-pages
  • statusz
    • statusz (structured)
  • flagz
    • flagz (structured)

statusz

Enabled using the ComponentStatusz feature gate, the /statusz endpoint displays high level information about the component such as its Kubernetes version, emulation version, start time and more.

The /statusz plain text response from the API server is similar to:

kube-apiserver statusz
Warning: This endpoint is not meant to be machine parseable, has no formatting compatibility guarantees and is for debugging purposes only.

Started: Wed Oct 16 21:03:43 UTC 2024
Up: 0 hr 00 min 16 sec
Go version: go1.23.2
Binary version: 1.32.0-alpha.0.1484+5eeac4f21a491b-dirty
Emulation version: 1.32.0-alpha.0.1484
Paths: /healthz /livez /metrics /readyz /statusz /version

statusz (structured)

Starting with Kubernetes v1.35, the /statusz endpoint supports a structured, versioned response format when requested with the appropriate Accept header. Without an Accept header, the endpoint returns the plain text response format by default.

To request the structured response, use:

Accept: application/json;v=v1beta1;g=config.k8s.io;as=Statusz

Example structured response:

{
  "kind": "Statusz",
  "apiVersion": "config.k8s.io/v1beta1",
  "metadata": {
    "name": "kube-apiserver"
  },
  "startTime": "2025-10-29T00:30:01Z",
  "uptimeSeconds": 856,
  "goVersion": "go1.23.2",
  "binaryVersion": "1.35.0",
  "emulationVersion": "1.35",
  "paths": [
    "/healthz",
    "/livez",
    "/metrics",
    "/readyz",
    "/statusz",
    "/version"
  ]
}

The config.k8s.io/v1beta1 schema for the structured /statusz response is as follows:

// Statusz is the config.k8s.io/v1beta1 schema for the /statusz endpoint.
type Statusz struct {
	// Kind is "Statusz".
	Kind string `json:"kind"`
	// APIVersion is the version of the object, e.g., "config.k8s.io/v1beta1".
	APIVersion string `json:"apiVersion"`
	// Standard object's metadata.
	// +optional
	Metadata metav1.ObjectMeta `json:"metadata,omitempty"`
	// StartTime is the time the component process was initiated.
	StartTime metav1.Time `json:"startTime"`
	// UptimeSeconds is the duration in seconds for which the component has been running continuously.
	UptimeSeconds int64 `json:"uptimeSeconds"`
	// GoVersion is the version of the Go programming language used to build the binary.
	// The format is not guaranteed to be consistent across different Go builds.
	// +optional
	GoVersion string `json:"goVersion,omitempty"`
	// BinaryVersion is the version of the component's binary.
	// The format is not guaranteed to be semantic versioning and may be an arbitrary string.
	BinaryVersion string `json:"binaryVersion"`
	// EmulationVersion is the Kubernetes API version which this component is emulating.
	// if present, formatted as "<major>.<minor>"
	// +optional
	EmulationVersion string `json:"emulationVersion,omitempty"`
	// MinimumCompatibilityVersion is the minimum Kubernetes API version with which the component is designed to work.
	// if present, formatted as "<major>.<minor>"
	// +optional
	MinimumCompatibilityVersion string `json:"minimumCompatibilityVersion,omitempty"`
	// Paths contains relative URLs to other essential read-only endpoints for debugging and troubleshooting.
	// +optional
	Paths []string `json:"paths,omitempty"`
}

flagz

Enabled using the ComponentFlagz feature gate, the /flagz endpoint shows you the command line arguments that were used to start a component.

The /flagz plain text response from the API server looks something like:

kube-apiserver flags
Warning: This endpoint is not meant to be machine parseable, has no formatting compatibility guarantees and is for debugging purposes only.

advertise-address=192.168.8.2
contention-profiling=false
enable-priority-and-fairness=true
profiling=true
authorization-mode=[Node,RBAC]
authorization-webhook-cache-authorized-ttl=5m0s
authorization-webhook-cache-unauthorized-ttl=30s
authorization-webhook-version=v1beta1
default-watch-cache-size=100

flagz (structured)

Starting with Kubernetes v1.35, the /flagz endpoint supports a structured, versioned response format when requested with the appropriate Accept header. Without an Accept header, the endpoint returns the plain text response format by default.

To request the structured response, use:

Accept: application/json;v=v1beta1;g=config.k8s.io;as=Flagz

Example structured response:

{
  "kind": "Flagz",
  "apiVersion": "config.k8s.io/v1beta1",
  "metadata": {
    "name": "kube-apiserver"
  },
  "flags": {
    "advertise-address": "192.168.8.4",
    "allow-privileged": "true",
    "anonymous-auth": "true",
    "authorization-mode": "[Node,RBAC]",
    "enable-priority-and-fairness": "true",
    "profiling": "true",
    "default-watch-cache-size": "100"
  }
}

The config.k8s.io/v1beta1 schema for the structured /flagz response is as follows:

// Flagz is the config.k8s.io/v1beta1 schema for the /flagz endpoint.
type Flagz struct {
	// Kind is "Flagz".
	Kind string `json:"kind"`
	// APIVersion is the version of the object, e.g., "config.k8s.io/v1beta1".
	APIVersion string `json:"apiVersion"`
	// Standard object's metadata.
	// +optional
	Metadata metav1.ObjectMeta `json:"metadata,omitempty"`
	// Flags contains the command-line flags and their values.
	// The keys are the flag names and the values are the flag values,
	// possibly with confidential values redacted.
	// +optional
	Flags map[string]string `json:"flags,omitempty"`
}

7 - Kubernetes Metrics Reference

Metrics (v1.36)

This page details the metrics that different Kubernetes components export. You can query the metrics endpoint for these components using an HTTP scrape, and fetch the current metrics data in Prometheus format.

List of Stable Kubernetes Metrics

Stable metrics observe strict API contracts and no labels can be added or removed from stable metrics during their lifetime.

List of Beta Kubernetes Metrics

Beta metrics observe a looser API contract than its stable counterparts. No labels can be removed from beta metrics during their lifetime, however, labels can be added while the metric is in the beta stage. This offers the assurance that beta metrics will honor existing dashboards and alerts, while allowing for amendments in the future.

List of Alpha Kubernetes Metrics

Alpha metrics do not have any API guarantees. These metrics must be used at your own risk, subsequent versions of Kubernetes may remove these metrics altogether, or mutate the API in such a way that breaks existing dashboards and alerts.