Mixed Version Proxy during Kubernetes 1.36 upgrades

Mixed Version Proxy during Kubernetes 1.36 upgrades

Kubernetes Mixed Version Proxy stops older API servers from handing back the wrong answer during a control plane upgrade. In a mixed-version control plane, that matters more than it sounds, because an old server can otherwise return a 404 for a resource that still exists on a newer peer. Kubernetes 1.36 moves the feature to Beta and enables it by default through UnknownVersionInteroperabilityProxy.

Why an older API server starts returning the wrong answer

The failure mode is dull and ugly. An older API server receives a request for an API it does not serve locally, checks its own view of the world, and replies as if the resource does not exist. That 404 is wrong, but it still looks clean enough to mislead clients, controllers, and anything trying to clean up after itself.

That matters during upgrades because discovery is part of normal API behaviour, not a special case. If discovery is incomplete on one server, a client can be pointed at the wrong place, or a controller can act on a false absence. That is how you end up with broken garbage collection or namespace deletion blocked by a server that simply gave up too early.

How the proxy path decides when to hand off a request

Mixed Version Proxy adds a peer lookup step before the server settles on a local 404. If the local API server does not have the resource, it checks discovery data for a peer that can serve it, then proxies the request instead of answering from its own limited view. The proxied request carries the x-kubernetes-peer-proxied header.

Discovery data from peers replaces the old StorageVersion lookup

The Beta implementation drops the old StorageVersion dependency for peer capability lookup and uses Aggregated Discovery instead. That matters because the StorageVersion API was never a good fit for CRDs or aggregated APIs. Those gaps are the sort that only surface during a control plane upgrade, when the wrong path suddenly becomes the only path.

With peer-aggregated discovery, API servers merge local and peer discovery data into one discovery document. That gives the older server enough information to find a newer peer that can answer the request. If the client asks for profile=nopeer, the server returns a local-only view instead, which keeps the peer data out of that specific discovery response.

Secure peer routing depends on the CA bundle and advertised address

Peer proxying is only useful if the TLS path works. --peer-ca-file is required for secure peer communication, and without it proxying fails because verification cannot be completed. In practice, that also means the server falls back to local APIs only, which brings the old 404 problem straight back.

Peer routing can use --peer-advertise-ip and --peer-advertise-port. If those are not set, the server uses --advertise-address or --bind-address as the fallback. That is fine until one of those addresses does not match the address peers should actually reach, which is the sort of detail that gets ignored right up until an upgrade starts shedding requests.

What changes in a 1.36 control plane upgrade

Kubernetes 1.36 turns Mixed Version Proxy into a default behaviour for the mixed-version control plane. The old server no longer has to pretend ignorance when a newer peer can serve the API. The result is fewer false 404s during the period when different API server versions are running side by side.

The practical change is in what discovery looks like and what headers appear on proxied requests. The discovery document can now include peer data, and the request path can show that proxying happened. That is the visible sign that the server took the handoff route rather than inventing a missing resource.

The headers and discovery view you will actually see

A proxied request carries x-kubernetes-peer-proxied. That is the easy marker that the local server did not answer directly.

Discovery also changes shape. When peer discovery is active, the API server presents a merged view of local and active peer APIs. If a client requests local-only discovery with profile=nopeer, it gets that narrower view instead of the merged one. That makes the discovery response less confusing during upgrades, which is rare enough to mention on its own.

The flags that need to be in place before the upgrade

UnknownVersionInteroperabilityProxy must be enabled, and in Kubernetes 1.36 it is enabled by default. The peer CA bundle still needs to be present, because --peer-ca-file is the switch that makes peer-aggregated discovery and secure proxying work. Without it, the server has no clean way to verify a peer.

For kubeadm-managed control planes, that typically means setting the relevant API server extra args in ClusterConfiguration, including peer-ca-file and the peer routing values when they are needed. apiServer.extraArgs.peer-ca-file: /etc/kubernetes/pki/ca.crt is the sort of line that looks boring until the upgrade depends on it.

The upgrade path is strict here. If peer discovery is not in place, the older server can still return a local 404 for something a newer peer knows about. Kubernetes has spent enough time learning that a wrong answer is worse than a delayed one.

Related posts

Weekly Tech Digest | 06 Jul 2026

Stay updated with the latest in tech! This digest covers AI ethics, auto industry shifts, and the impact of politics on technology, exploring today's pressing issues.

wolfCOSE zero-allocation parsing in embedded C

wolfCOSE looks sensible only if you care about what your firmware actually has to carry. I like that, because on small targets the wrong crypto feature can cost more than the message itself, and there...

restic | v0.19.1

restic v0 19 1: safer FUSE mounts and mountpoint checks, robust backup source and exclude handling, clearer CLI JSON output, Windows SFTP deletion fixes