controller-runtime cache: stale reads and conflicts

controller-runtime cache: stale reads and conflicts

controller-runtime reads are cheap because they usually come from a local in-memory cache, not straight from kube-apiserver. That cache is useful, but it is not strongly consistent after a write, and that catches people out when they expect an update to be visible immediately. The result is stale reads, occasional conflicts, and the odd controller that looks haunted for a few seconds.

Why the cache returns old objects after a write

A manager starts by listing objects and then keeps its cache current through watch events. In normal reconciler code, r.Get() and r.List() hit that local store, while r.Update() goes straight to the API server. That split keeps reads fast and reduces load on the control plane, but it also means the cache can lag behind the last write.

A common failure looks like this: read an object, change it, update it, then read it again and still see the old state. The update has gone to the API server, but the informer-backed cache has not processed the watch event yet. For a short window, the reconciler is looking at stale data and may make the wrong next decision.

The same model applies to resourceVersion. Kubernetes uses it for optimistic concurrency, so an update with a mismatched version comes back as a 409 Conflict. That is normal behaviour, not a random failure. It usually means another writer changed the object first, or the object you read was already stale when you tried to write it.

Where conflict errors come from in update paths

Conflict errors are part of the update contract, not an edge case to ignore. The API server checks that the resourceVersion you send still matches the version stored in etcd. If it does not, the write is rejected. That protects the object from silent overwrites, which is useful until your reconcile loop assumes it has the latest copy and tries to patch it anyway.

This matters most when code reads from the cache, mutates the object, and writes it back in the same path. If another controller or another reconcile run updates the same object in between, the write fails with 409 Conflict. The fix is not to keep retrying blind. Re-read, merge the change you actually need, and try again with the new version.

resourceVersion also drives watch resumption, so it is not just an update token. It is part of how Kubernetes keeps list-watch state moving without starting from zero every time. That is one reason cache-backed reads and server-side writes fit together, even when they are not perfectly in step.

Bypassing the cache when you need a live read

APIReader exists for the cases where the cache is the wrong tool. It bypasses the shared informer cache and reads directly from the API server. That gives a live view, which is useful when you need a strongly consistent read after a write, or when a specific decision cannot tolerate stale state.

It is not something to sprinkle through a controller because the cache feels awkward. Direct reads are more expensive, and a controller that fires hundreds of them per second will load the API server and etcd for no good reason. Use APIReader for narrow cases where the local cache cannot give the answer you need, then go back to the cache for ordinary reconciler flow.

Indexing matters here too. A badly written List() can quietly turn into a linear scan across every cached object of that type. That is fine with a handful of objects and miserable with tens of thousands. IndexField and MatchingFields exist so selected lookups can use an index instead of walking the whole store. Without an index, the cache still works, just with the sort of efficiency that keeps a node fan busy for no real benefit.

The cache is not free memory either. Large object sets can quietly consume gigabytes if the manager is holding more than expected or if the watched types are broad. SharedIndexInformer, Reflector, and Indexer make the reads cheap, but they also keep data in process so the controller can answer queries without hitting the API server every time. That trade-off is the whole point, and it is also the reason cache design matters as much as reconcile logic.