Static Pod config breaks without API access

What you see

The failure usually shows up as a pod that never settles, or one that disappears after a restart and never comes back cleanly. The important part is that the breakage happens before the container gets far enough to be useful. Static Pods are read by kubelet from the local manifest, so the node enforces the rule even if the API server is healthy.

The problem is not limited to one field name. Any API-backed reference hidden in the pod specification can trigger the failure. That includes volume sources and env-related references that expect the API server to fill in data the node does not have locally.

Where it happens

configMapRef and secretRef stop working on the node. Kubelet no longer allows a Static Pod to depend on API objects for those values, and the old feature gate for allowing that behaviour has gone.

The error text usually points back to kubelet rather than the API server. Expect a rejection that mentions Static Pod API references being prohibited. If the pod used to start and now fails after a node upgrade, that is the first place to look.

Node logs are the cleanest signal. The API server may still be up and responding, which makes this look like a cluster issue at first glance. It is not. It is a manifest issue that kubelet now refuses to ignore.

Find the cause

Check the manifest, not the API server. Static Pods are still defined by local files, so the bad reference lives in the pod specification on disk. If a Static Pod reaches for a ConfigMap or Secret, the field needs to be removed or replaced.

Look for any API-backed fields in the pod specification, especially where a volume, env var, or image-related setting expects cluster data. A manifest that was acceptable in older releases can now fail simply because it assumes kubelet can fetch something from the API server.

Confirm the node is running the new rule by checking the kubelet behaviour after a restart or manifest reload. If the same file keeps failing and the logs complain about API references, the node is doing exactly what it should.

Fix

Replace API references with local files or other node-side input. Static Pods need data that kubelet can read without talking to the API server. That usually means host-mounted files, local config, or another source that exists on the node before the pod starts.

Keep the fix narrow. Do not swap one reference for another that still depends on cluster state. The pod has to survive on node-local input alone, because kubelet will not make an exception for Static Pods here.

Check it’s fixed

Reload the Static Pod and watch kubelet state, not just pod status. The manifest should be picked up again, the rejection should stop, and the pod should move back to a running state without the API-backed reference in place.

Check the node logs one more time after the reload. If the same config path is still being blocked, the manifest still contains a hidden API reference. Once that is gone, kubelet has no reason to refuse the pod.

Related posts

k3s | v1.36.4+k3s1

k3s update: Kubernetes and Go bumps, Traefik chart renames kubernetesIngressNginx to kubernetesIngressNGINX, component updates and maintenance

k3s | v1.36.4+k3s1

k3s v1 36 4 k3s1: Kubernetes v1 36 4 and Go 1 26 7, Traefik chart v40 and ingress nginx provider rename, embedded component bumps, test in staging

n8n | n8n@2.36.8

n8n 2 36 8: Fixes domain restricted credential bug so nodes can use credentials scoped to domain, no breaking changes, upgrade if you use domain restrictions