I build Kubernetes clusters to survive real faults, not to win configuration contests. This is what I do during setup to keep them standing when something fails, and the places I see people cut corners if they are rushing. Keep it concrete. Test it, then test the restore.
Control plane and data store
Run an odd number of control-plane nodes, usually three. That gives etcd a majority quorum without pushing the cost up too far. Put an external load balancer in front of the API servers so the kube-apiserver endpoints stay reachable when a control-plane node restarts.
Back up etcd on a schedule and test the restore on a spare machine, not just once. I keep backups off-node and hold at least three days of history. For etcd, use TLS and client authentication, and rotate certificates before they expire.
Keep CoreDNS as a Deployment with two or three replicas. Set resource requests and limits for control-plane components so they are not competing with user pods. Use PodDisruptionBudgets for critical kube-system pods so rolling upgrades do not take out the whole control plane.
If you use kubeadm, follow its HA patterns. Stacked etcd is the simpler option for small clusters. External etcd makes more sense for larger or multi-site setups.
Worker nodes and workloads
Make worker nodes predictable. Label and taint nodes by role: compute, storage, GPU, whatever fits the workload. Use node selectors or node affinity in pod specs rather than relying on hand-waving and luck.
Set resource requests and limits on every deployment. Use LimitRanges and ResourceQuotas per namespace so one noisy app does not starve everything else. Use liveness and readiness probes on every container: readiness controls rollout, liveness restarts stuck processes.
For anything you care about, run at least two replicas. One replica is a single point of failure with nicer formatting.
For stateful services, prefer StatefulSets with a storage class that supports reclaimPolicy and snapshots. For persistent data, use volume snapshots or external storage with multi-AZ replication if you run across zones. Avoid local disks for critical data unless you already have a proper backup plan.
Network and security
Choose a CNI you understand, and run it with two or more replicas where that applies. Monitor CNI metrics and test network partition scenarios in a lab.
Keep kube-proxy in IPVS mode for performance if the traffic pattern justifies it.
Apply RBAC with least privilege. Create service accounts for controllers and give them only the ClusterRoles they need. Enable audit logs and ship them to a central location; they are not much use if they disappear with the node.
Use Pod Security admission controls, or whatever your cluster version supports, to limit hostPath, privileged, and hostNetwork use. Use network policies to block lateral movement inside the cluster. Start with default-deny and open ports per namespace as needed.
Keep registry credentials in Secrets and scan images for CVEs before you deploy them.
Operate it like it will fail
Automate upgrades on a staggered schedule, and practise the upgrade path in a non-production cluster first. Run synthetic transaction checks so you know when an app is broken in practice, not just when pods are failing.
Alert on high API server latency, etcd commit duration, and node heartbeat failures. Backups need more than faith: perform a restore at least quarterly.
Write short runbooks for common failures: API server unresponsive, etcd degraded, node unreachable. Test cluster resilience by simulating a control-plane node loss, a full-node disk failure, and a network partition. Track configuration changes in Git and apply them with GitOps tooling so you can roll back quickly.
My list is still the same: three control-plane nodes, tested etcd backups, resource requests, probes, RBAC, network policies, and rehearsed restores. Get those right and the cluster has a chance when real faults turn up.
