1. What Happened During a Late-Night Maintenance Window — Pods Stopped Scaling After a Live Migration
It's not an uncommon report: right after live-migrating a K3s control plane VM on Proxmox VE, the cluster's etcd quorum collapses for a few minutes and pod scheduling grinds to a complete halt. The intent was simply to evacuate the VM for a host firmware update, but suddenly kubectl get pods stops responding, load is climbing, and the HPA does nothing at all. Looking at the logs after recovery, the etcd leader was simply absent for those few minutes.
What makes this especially tricky is that neither Proxmox nor K3s tends to surface an explicit "this broke" error. The live migration completes exactly as specified, and K3s self-heals after a while, so the incident gets written off as "just a temporary blip" — and the same failure recurs. This article breaks down why this combination can fail structurally, looking at both Proxmox's virtualization mechanics and etcd's consensus algorithm.
2. Why a "Zero-Downtime" Live Migration Can Still Break etcd
Proxmox VE's High Availability feature manages the entire cluster through Corosync-based quorum voting, and when a node fails, the HA Manager automatically restarts resources on another node. Live migration itself works by transferring memory contents over the network before cutting over, which looks safer than a relocate that stops and restarts the VM — but the transfer temporarily strains the host's network bandwidth and CPU scheduling.
Meanwhile, K3s's HA setup with embedded etcd is designed to maintain quorum (n/2)+1 across an odd number of server nodes (minimum three). etcd elects its leader using the Raft consensus algorithm, and its liveness detection depends on extremely tight timing. According to etcd's official tuning guide, the default heartbeat-interval is 100ms and the election-timeout is 1000ms. In other words, if a heartbeat that's supposed to arrive every 100ms goes missing for more than 1000ms, followers conclude "the leader is dead" and trigger a new election.
If the control plane VM's network gets slightly congested during live migration, or vCPU scheduling gets delayed, that grace period can be exceeded without much effort. The virtualization layer's notion of "zero downtime" and the distributed consensus algorithm's notion of "liveness confirmation" are simply designed around entirely different timescales.
3. Another Blind Spot: Disk I/O — fsync Latency and etcd Stability
Timeout problems aren't limited to the network — disk I/O causes them too. etcd is designed so that it cannot commit a Raft log entry until it has been synchronously written to disk (fsync). etcd's official hardware recommendations call for at least 500 IOPS of disk performance for busy clusters, and explicitly state that "if writes take too long, heartbeats will time out, triggering elections that undermine cluster stability." Disk latency translates directly into leader liveness decisions.
Proxmox's live migration itself is mostly about memory transfer, but if you're running on a slow virtual disk that contends with shared storage or other VMs, fsync latency tends to degrade from the I/O spike before and after migration. In fact, a CNCF blog post notes that many production etcd incidents are triggered by degraded disk I/O latency or member-to-member RTT, and that pinpointing the root cause requires continuously monitoring WAL fsync duration.
4. Proxmox-Side Countermeasures — Negative Affinity and Storage Requirements
The first thing to address is placement design on the Proxmox side. Proxmox VE can control VM placement through resource affinity rules, and a negative affinity setting such as ha-manager rules add resource-affinity keep-separate can force K3s control plane VMs onto physically separate hosts. If all three control plane VMs happen to sit on the same physical host, maintenance on that single host is enough to break etcd quorum (which requires at least 2 of 3 nodes).
On the storage side, assign the control plane VMs a dedicated fast block device to avoid I/O contention on shared storage, and monitor to ensure failover latency doesn't significantly exceed 10ms. It also helps to run etcdctl endpoint status --cluster before a migration to identify the current etcd leader, and establish an operational rule of migrating the leader VM last (or transferring leadership beforehand) — a small step that can meaningfully shorten the impact window.
This kind of affinity design and storage sizing is exactly the sort of "hidden prerequisite" that's easy to overlook when self-hosting K3s. Kubo On-Premise can deliver fully managed Kubernetes on top of on-premises virtualization platforms like Proxmox, with control plane placement design and storage requirements built in as standard configuration.
5. etcd-Side Countermeasures — Tuning Heartbeat/Election-Timeout to Real-World Measurements
Another countermeasure is tuning etcd's own timeout parameters to match the measured characteristics of the virtualization platform. etcd's tuning guide recommends setting the election-timeout to at least 10x the round-trip time between members, and in environments with unstable fsync latency, that multiplier should be pushed even higher. Running with default values on top of a virtualization platform leaves too little tolerance for the transient delays that occur during live migration.
On the monitoring side, it's worth continuously checking whether the p99 of wal_fsync_duration_seconds exceeds 10ms, and alerting when it does. This isn't unique to virtualization platforms — even large-scale cluster management tools like kops have reported cluster outages caused by etcd quorum loss, which is further evidence that etcd's consensus mechanism itself is sensitive to transient I/O and network disruptions regardless of the underlying platform.
Designing and continuously operating this kind of parameter tuning and metrics monitoring on your own isn't a side project. With Kubo Cloud's Captain UI, control plane health metrics are visualized out of the box, so you don't have to keep re-deriving the tuning know-how from scratch.
6. Recovery Steps When It Does Break, and Operational Rules to Prevent It
Even with countermeasures in place, quorum can still break if you perform operations that affect multiple control plane VMs at once — a network change or a bulk migration, for example. There are reports of clusters where multiple K3s nodes were brought up simultaneously during a network segment change, and the control plane went down with an error like etcd: cluster ID mismatch. Recovering from this kind of failure involves resetting one node at a time as a single-node etcd cluster, then rejoining the rest in sequence.
The lesson is simple: never apply changes that affect multiple etcd members at once — network changes, live migrations, OS updates, and so on. Thoroughly enforcing a one-node-at-a-time operational rule, checking state after each step before moving to the next, avoids the majority of failures of this kind. When Proxmox itself loses quorum, the HA Manager can't reset its watchdog and triggers automatic fencing after 60 seconds, so it's also worth revisiting redundant network paths and using a QDevice.
If you're distributing access to the control plane with simple DNS-based load balancing, that's another thing to watch. A K3s community discussion points out that static DNS records with no health checks can introduce unexpected startup delays, and recommends switching to a health-check-aware load balancer instead.
Conclusion — Treat "Virtualization Platform Assumptions" and "Distributed Consensus Assumptions" as Separate Design Concerns
Proxmox's live migration is an availability feature of the virtualization layer, while K3s's embedded etcd is a consensus mechanism for the Kubernetes cluster layer. These two are designed at separate layers, on separate timescales, and making one look "zero-downtime" doesn't stop it from exceeding the other's liveness-detection timeout. The fix needs to be built across three layers: affinity design and storage requirements on the Proxmox side, timeout tuning and monitoring on the etcd side, and an operational rule against simultaneous changes.
Choosing self-hosted K3s to avoid the high cost and lock-in of EKS or AKS is a perfectly reasonable call. But that operational cost includes the less visible burden of discovering and redesigning around host-to-etcd assumption clashes like this one, on your own. Kubo keeps the full power of K3s-based Kubernetes intact while building in this kind of availability design as standard configuration. If you're already dealing with a similar failure, or planning to run a K3s HA cluster on Proxmox in production, reaching out to discuss your setup is one option worth considering. Our pricing page also lets you compare costs against EKS/AKS.