Detecting node contention with Pressure Stall Information
Pressure Stall Information records time spent waiting for resources rather than just time spent busy. CPU saturation, memory reclaim, and I/O pressure show up as stalled time, which is more useful when pod latency spikes before node averages look ugly. That makes Kubernetes PSI metrics a better fit for spotting node resource contention than raw usage charts.
PSI exposes waiting time that CPU usage misses
CPU percentage tells you how much work ran. PSI tells you how long tasks waited to run. Those are not the same thing, and the gap matters when a node gets twitchy under bursty load.
A node can show moderate CPU use while a run queue builds, short bursts hit the scheduler, and containers start missing latency targets. Memory pressure behaves in a similar way. A pod may not use much RAM overall, yet still stall while the kernel reclaims memory or waits on page faults. I/O pressure is the same story with disks and volumes. The node is not “busy” in the neat dashboard sense, but it is holding work up.
Read the 10, 60, and 300 second windows against bursty load
PSI reports moving averages over 10, 60, and 300 seconds. Use the short window for spikes, the mid window for sustained contention, and the long window for the ugly stuff that never quite clears. A flat 300 second line with noisy 10 second bursts usually points to brief congestion. If the 60 and 300 second windows rise together, the node is spending a real chunk of time stalled.
That is the sort of pattern worth watching before an SLO breach turns into an argument.
Tie stalled time back to CPU, memory, and I/O pressure
CPU PSI is the obvious one, but memory and I/O often explain the mess better. A latency spike on an otherwise quiet node can come from memory reclaim, not CPU saturation. Slow writes, noisy neighbours, or a stressed volume path can sit behind I/O PSI even when CPU charts look bored.
Treat the three PSI dimensions as separate failure modes. Mixing them together hides the cause and makes the alert useless.
Where Kubernetes surfaces PSI and where it does not
Kubernetes exposes PSI through kubelet on Linux nodes that support it. The practical routes are the /metrics/cadvisor endpoint and the Summary API, with per-container cpu.psi, memory.psi, and io.psi fields available there. Prometheus can scrape the cAdvisor path directly.
Unsupported nodes do not behave like low-pressure nodes. They simply do not have valid PSI data. That distinction matters because a zero can mean “nothing stalled” or “nothing was collected”. Only one of those should drive paging.
Scrape /metrics/cadvisor or pull it through the Summary API
If Prometheus already scrapes kubelet, PSI fits into the same collection path through /metrics/cadvisor. The Summary API also carries PSI data, and it is handy when a script or debugging session needs a quick container-level view. kubectl plus jq is enough to inspect the fields without building another dashboard just to prove the node is miserable.
Avoid proxying kubelet casually through the API server. It is a privileged route, and handing it out to the wrong people is a neat way to make a bad day worse.
Treat missing metrics on unsupported nodes as missing, not zero
Kubernetes v1.36 omits PSI metrics on nodes where support is not present. That is the correct behaviour. A missing metric is a clear signal that the host cannot report PSI. A zero is a lie on older setups and a trap for alerting rules.
Prometheus rules need to handle this cleanly. Filter out unsupported hosts rather than comparing them with real PSI-capable nodes. A flat zero across the board on mixed hardware is usually a collection problem with better PR than the actual problem.
Turning PSI into a node contention signal
PSI becomes useful when it is tied back to service behaviour. A node with rising CPU or memory stall time and matching pod latency is telling a simple story. It is under contention and the scheduler, kernel, or storage path is making work wait.
The useful alert is not “PSI is non-zero”. That is too blunt to matter. The useful alert is a pattern: short-window stall time rising on a node that also shows latency, restarts, or queue growth in pods that live there. That gives a readable signal instead of noise dressed up as observability.
Watch for contention patterns that line up with pod latency
Look for stall spikes that line up with request latency, timeout bursts, or container throttling. CPU PSI often shows scheduler pressure before outright saturation. Memory PSI often appears before OOM kills, which is a far nicer time to notice a node is tight. I/O PSI shows up when the node starts waiting on storage rather than doing useful work.
A single PSI blip is not much. Repeated blips across several minutes, or a rising 60 second average, is a different matter. That is where node resource contention starts to look like a service issue.
Use Prometheus alerts that ignore flat zeroes on unsupported hosts
Prometheus needs a rule that accepts missing PSI as missing. Do not alert on zero where the node has no PSI support. Do alert on rising averages over supported Linux nodes when the stall pattern matches latency or saturation elsewhere.
A sensible rule keeps unsupported hosts out of the expression entirely. Otherwise a mixed cluster turns into a false-positive factory, which is a grim hobby.
Validating it on real nodes before you trust it
PSI is only useful when the kernel, cgroup setup, and kubelet all agree. Linux kernel 4.20 or later is required. Nodes need cgroup v2, CONFIG_PSI=y, and the kernel must not be booted with psi=0. Some images, including COS, default to PSI being enabled, which saves a bit of faff.
Kubernetes v1.36 makes PSI generally available through kubelet without a feature gate, after alpha in v1.33 and beta in v1.34. That does not help if the node cannot actually emit the data. A friendly dashboard built on broken assumptions still lies with a straight face.
Check cgroup v2, kernel PSI support, and kubelet version
Start with the node itself. Confirm cgroup v2 is in use, PSI support exists in the kernel, and kubelet is new enough to expose the fields. Windows nodes do not support PSI, so leave them out of any rule that assumes Linux telemetry exists.
Version checks are boring, but missing support is the kind of bug that survives all the way to an incident review if nobody asks the dull questions.
Test the dump path, not the policy wording
Read the actual /metrics/cadvisor or Summary API output and check for cpu.psi, memory.psi, and io.psi. Do not trust a policy or a dashboard label until the values are visible on a live node. If the fields are absent, treat them as absent. If they are present, compare them with real pod behaviour and see whether the stall pattern matches the latency you care about.
That is the part that stops PSI from becoming another pretty graph nobody believes.
