[{"data":1,"prerenderedAt":541},["ShallowReactive",2],{"blog-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending":3,"blog-related-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending":484,"blog-en-k3s-local-path-provisioner-capacity-node-affinity-pvc-pending-alt":471},{"id":4,"title":5,"author":6,"body":7,"date":465,"description":466,"extension":467,"image":468,"locale":469,"meta":470,"navigation":471,"path":472,"seo":473,"stem":474,"tags":475,"__hash__":483},"blog\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending.md","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","Kubo Team",{"type":8,"value":9,"toc":448},"minimark",[10,15,23,35,47,50,54,60,79,95,98,103,128,135,144,148,154,160,179,185,191,195,216,224,228,234,237,260,269,285,289,304,308,314,317,330,342,351,371,374,378,384,394,397,423,432,439],[11,12,14],"h2",{"id":13},"_1-one-day-a-k3s-pod-simply-stopped-coming-back-from-pending","1. One Day, a K3s Pod Simply Stopped Coming Back From Pending",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection01.webp",[16,24,25,26,30,31,34],{},"You created a new PVC on your K3s cluster, but the Pod sits in ",[27,28,29],"code",{},"Pending"," for tens of minutes and never reaches ",[27,32,33],{},"Running",". If you've run K3s in production for any length of time, you've probably hit this at least once.",[16,36,37,38,41,42,46],{},"Running ",[27,39,40],{},"kubectl describe pod"," shows nothing wrong with image pulls or scheduling itself. In practice, the culprit is often something quieter: ",[43,44,45],"strong",{},"the node's local disk capacity had been silently filling up."," In architectures that lean on node-local storage, like K3s typically does, this kind of invisible capacity trap is a recurring source of production incidents.",[16,48,49],{},"This article works through four commonly overlooked aspects of K3s local storage design, in order: a design constraint in K3s's default storage mechanism (local-path-provisioner), hypervisor-side thin provisioning on platforms like Proxmox, the Volume Node Affinity Conflict that appears when nodes change, and the difference between a snapshot and an actual backup.",[11,51,53],{"id":52},"_2-local-path-provisioner-doesnt-actually-look-at-pvc-capacity","2. local-path-provisioner Doesn't Actually Look at PVC \"Capacity\"",[16,55,56],{},[19,57],{"alt":58,"src":59},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection02.webp",[16,61,62,63,70,71,78],{},"K3s ships with Rancher's ",[64,65,69],"a",{"href":66,"rel":67},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner",[68],"nofollow","Local Path Provisioner"," out of the box. By default, persistent volumes are created on the node's own filesystem, under ",[64,72,75],{"href":73,"rel":74},"https:\u002F\u002Fdocs.k3s.io\u002Fadd-ons\u002Fstorage",[68],[27,76,77],{},"\u002Fvar\u002Flib\u002Francher\u002Fk3s\u002Fstorage",". It's a convenient way to get PersistentVolumes working with almost no setup — but it comes with a design constraint that's easy to miss.",[16,80,81,84,85,88,89,94],{},[43,82,83],{},"local-path-provisioner does not actually enforce the capacity you declare on a PVC."," This is acknowledged directly in discussion on Rancher's own GitHub repository. One issue thread notes that \"since local-path-provisioner doesn't enforce capacity limitations, there's no reason to throw errors\" when someone changes ",[27,86,87],{},"spec.capacity"," — ",[64,90,93],{"href":91,"rel":92},"https:\u002F\u002Fgithub.com\u002Francher\u002Flocal-path-provisioner\u002Fissues\u002F190",[68],"confirming that capacity limits are effectively ignored by design",".",[16,96,97],{},"In other words, writing \"10Gi\" on a PVC is just a declaration at the Kubernetes API level. If a container keeps writing more data than that, local-path-provisioner won't stop it. The data keeps consuming the node's physical disk, eventually squeezing out space that other Pods depend on.",[99,100,102],"h3",{"id":101},"what-actually-happens","What Actually Happens",[104,105,106,114,121],"ul",{},[107,108,109,110,113],"li",{},"An application can keep writing logs or cache data past its ",[27,111,112],{},"requests.storage"," value without ever hitting an error",[107,115,116,117,120],{},"As node disk usage approaches 100%, the kubelet raises ",[27,118,119],{},"DiskPressure"," on the node, which starts evicting existing Pods and refusing to schedule new ones",[107,122,123,124,127],{},"The overrun never shows up in the PVC's status. ",[43,125,126],{},"What you actually need to watch is per-node disk usage"," — the PVC's declared value alone won't warn you",[16,129,130,131,134],{},"Volume Expansion (",[27,132,133],{},"AllowVolumeExpansion",") support has also long been deferred in local-path-provisioner, so in practice you have to assume capacity management is something you monitor and provision for at the node level.",[16,136,137,138,143],{},"Closing this gap between declared and actual usage with a homegrown monitoring dashboard is tedious work. With ",[64,139,142],{"href":140,"rel":141},"https:\u002F\u002Fkubo.hexabase.io\u002F",[68],"Kubo","'s Captain UI, per-node resource status is visible directly from the cluster management screen, which makes it much easier to spot these capacity warning signs early.",[11,145,147],{"id":146},"_3-the-overlooked-second-factor-hypervisor-side-thin-provisioning","3. The Overlooked Second Factor: Hypervisor-Side Thin Provisioning",[16,149,150],{},[19,151],{"alt":152,"src":153},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection03.webp",[16,155,156,157],{},"If your K3s nodes are VMs running on a hypervisor like Proxmox VE, things get another layer more complicated. On top of K3s ignoring capacity, ",[43,158,159],{},"the hypervisor's own storage can have a nearly identical \"invisible capacity\" problem.",[16,161,162,163,168,169,174,175,178],{},"Proxmox VE's default storage backends, ZFS and LVM-Thin, both support thin provisioning. According to ",[64,164,167],{"href":165,"rel":166},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage:_LVM_Thin",[68],"Proxmox's official wiki",", with this approach \"volumes can be much larger than physically available space.\" Similarly, ",[64,170,173],{"href":171,"rel":172},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage:_ZFS",[68],"the official ZFS storage documentation"," explains that enabling the ",[27,176,177],{},"sparse"," property creates sparse volumes \"whose reservation is not equal to the volume size.\"",[16,180,181,182],{},"In other words, even if you assign a 100GB virtual disk to each of several K3s node VMs, the pool only actually consumes physical capacity equal to the data that has really been written. That's an efficient design — but it also means ",[43,183,184],{},"multiple VMs sharing the same storage pool can easily create an \"overcommit\" state, where the sum of declared virtual capacity exceeds the pool's physical capacity.",[16,186,187,188],{},"When the thin pool's physical capacity actually runs out, writes start failing abruptly, even though the VM (the K3s node) still appears to have free disk space. With K3s's local-path-provisioner not watching capacity, and the hypervisor's pool exhaustion being just as hard to notice, it's a natural consequence that ",[43,189,190],{},"the failure develops while remaining invisible on two separate layers at once.",[99,192,194],{"id":193},"practical-mitigations","Practical Mitigations",[104,196,197,204,213],{},[107,198,199,200,203],{},"Monitor hypervisor storage pool usage ",[43,201,202],{},"at the pool level",", not per-VM",[107,205,206,207,212],{},"As ",[64,208,211],{"href":209,"rel":210},"https:\u002F\u002Fpve.proxmox.com\u002Fwiki\u002FStorage",[68],"Proxmox's storage overview page"," notes, ZFS and LVM-Thin are node-local, non-shared storage; if you need live migration or redundancy across multiple nodes, moving to distributed storage like Ceph\u002FRBD is an option worth considering",[107,214,215],{},"Design capacity monitoring for K3s and the hypervisor as two independent concerns",[16,217,218,219,223],{},"Tracking when to rebuild a node or relocate a VM, and which PV is pinned to which node, by hand is a recipe for tribal knowledge. ",[64,220,222],{"href":140,"rel":221},[68],"Kubo Cloud"," ships with a Rancher-based management layer as standard, which makes node-to-volume relationships visible and helps prevent this class of incident from recurring.",[11,225,227],{"id":226},"_4-crossing-nodes-breaks-everything-the-real-nature-of-volume-node-affinity-conflict","4. Crossing Nodes Breaks Everything: The Real Nature of Volume Node Affinity Conflict",[16,229,230],{},[19,231],{"alt":232,"src":233},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection04.webp",[16,235,236],{},"There's another classic dead end with node-local storage: the \"Volume Node Affinity Conflict.\"",[16,238,239,240,243,244,249,250,253,254,259],{},"The ",[27,241,242],{},"local"," volume type defined in the ",[64,245,248],{"href":246,"rel":247},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fvolumes\u002F",[68],"official Kubernetes documentation"," is, by nature, storage that \"exists only on a specific node,\" and a PersistentVolume using it ",[43,251,252],{},"requires a node affinity (nodeAffinity) specification."," ",[64,255,258],{"href":256,"rel":257},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fstorage\u002Fpersistent-volumes\u002F",[68],"The PersistentVolume documentation"," likewise lists Node Affinity as one of the key configuration fields for a volume.",[16,261,262,263,266,267,94],{},"The flip side of this is that ",[43,264,265],{},"once a PV is tied to a given node, a Pod using it can only be scheduled on that same node."," If you relocate a VM to a different physical host on Proxmox, or rebuild a K3s node from scratch, disk paths or UUIDs can change — and the mismatch between the PV's declared node affinity and the Pod's actual scheduling target leaves the Pod stuck in ",[27,268,29],{},[16,270,271,272,277,278,281,282],{},"This constraint has been a long-standing operational pain point, but there's an interesting development worth noting. According to the ",[64,273,276],{"href":274,"rel":275},"https:\u002F\u002Fwww.kubernetes.io\u002Fblog\u002F2026\u002F01\u002F08\u002Fkubernetes-v1-35-mutable-pv-nodeaffinity\u002F",[68],"Kubernetes blog"," published in January 2026, ",[43,279,280],{},"Kubernetes v1.35 added the ability to make a PersistentVolume's node affinity mutable, as an alpha feature."," The post states plainly that \"this field was previously immutable, and it is now mutable in Kubernetes v1.35 (alpha),\" explaining that this lets administrators update node affinity as underlying storage changes (for example, migration to regional disks). Read the other way, ",[43,283,284],{},"this change is itself evidence of how real a problem the immutability of PV node affinity has been in practice.",[99,286,288],{"id":287},"practical-workarounds","Practical Workarounds",[104,290,291,294,301],{},[107,292,293],{},"If you use node-local PVs, build PV recreation into your operational workflow for VM relocation and node replacement",[107,295,296,297,300],{},"Use ",[27,298,299],{},"volumeBindingMode: WaitForFirstConsumer"," on your StorageClass so the PV only binds after the Pod's scheduling target is decided, reducing unnecessary mismatches",[107,302,303],{},"For workloads that genuinely need portability across nodes, consider not using node-local storage at all",[11,305,307],{"id":306},"_5-a-zfs-snapshot-is-not-a-backup","5. A ZFS Snapshot Is Not a \"Backup\"",[16,309,310],{},[19,311],{"alt":312,"src":313},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection05.webp",[16,315,316],{},"Even once you've worked through capacity and scheduling issues, there's one more misconception worth untangling: the belief that \"as long as I have a snapshot, my backup story is complete.\"",[16,318,319,320,325,326,329],{},"A ZFS snapshot, according to the ",[64,321,324],{"href":322,"rel":323},"https:\u002F\u002Fopenzfs.github.io\u002Fopenzfs-docs\u002Fman\u002Fmaster\u002F8\u002Fzfs-snapshot.8.html",[68],"official OpenZFS documentation",", is defined as \"a consistent image of a dataset at a specific point in time... that includes all modifications to the dataset made by system calls that have successfully completed before that point in time.\" That's a powerful capability, but it remains ",[43,327,328],{},"a point-in-time copy at the storage layer"," — it does not protect the state of your Kubernetes cluster itself (the resource definitions stored in etcd).",[16,331,332,333,338,339],{},"Kubernetes cluster state lives in ",[64,334,337],{"href":335,"rel":336},"https:\u002F\u002Fetcd.io\u002Fdocs\u002Fv3.5\u002Fop-guide\u002Frecovery\u002F",[68],"etcd",", and the official documentation explains that \"etcd v3 provides snapshot and restore facilities to recreate the cluster without v3 key data loss\" for disaster recovery. In other words, an etcd snapshot ",[43,340,341],{},"protects your cluster's configuration (Deployments, Services, Secrets, and so on) — it does not protect the actual data sitting in your PVCs\u002FPVs.",[16,343,344,345,350],{},"Meanwhile, the ",[64,346,349],{"href":347,"rel":348},"https:\u002F\u002Fvelero.io\u002Fdocs\u002Fmain\u002Fhow-velero-works\u002F",[68],"official Velero documentation"," explains that Velero archives Kubernetes objects to object storage while also, when configured, \"calling the cloud provider API to make disk snapshots of persistent volumes.\" Velero is therefore a broader protection layer that can cover both API resources and PV data.",[104,352,353,359,365],{},[107,354,355,358],{},[43,356,357],{},"Hypervisor snapshot"," (e.g., a Proxmox snapshot): a point-in-time copy of the VM disk. Useful for recovering a single K3s node, but it doesn't guarantee consistency across the whole cluster",[107,360,361,364],{},[43,362,363],{},"etcd snapshot",": protects cluster configuration. Does not cover the actual data in PVCs\u002FPVs",[107,366,367,370],{},[43,368,369],{},"Velero",": can cover both API resources and PV data, but requires its own separate setup and operation",[16,372,373],{},"These three layers play different roles. Relying on just one of them is exactly how you end up discovering, mid-incident, that the thing you actually needed to restore isn't what you backed up.",[11,375,377],{"id":376},"_6-summary-what-to-check-before-you-trust-node-local-storage","6. Summary: What to Check Before You Trust Node-Local Storage",[16,379,380],{},[19,381],{"alt":382,"src":383},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Fsection06.webp",[16,385,386,387,389,390,393],{},"A K3s PVC stuck in ",[27,388,29],{}," rarely comes down to a simple misconfiguration. More often, the real cause is the fact that ",[43,391,392],{},"local-path-provisioner was never designed to watch capacity"," in the first place. Layer in hypervisor thin provisioning, node affinity constraints, and misunderstandings about backups, and the root cause becomes even harder to pin down.",[16,395,396],{},"Here's a quick operational checklist covering the four points this article walked through:",[104,398,399,405,411,417],{},[107,400,401,404],{},[43,402,403],{},"Capacity monitoring",": Are you watching per-node disk usage, rather than trusting the PVC's declared capacity?",[107,406,407,410],{},[43,408,409],{},"Thin provisioning monitoring",": Are you checking the hypervisor's storage pool at the pool level, not just per-VM usage?",[107,412,413,416],{},[43,414,415],{},"Node affinity operations",": Does your operational workflow assume that rebuilding a node or relocating a VM can strand a PV?",[107,418,419,422],{},[43,420,421],{},"Three-layer backups",": Do you understand the different roles of hypervisor snapshots, etcd snapshots, and Velero (including PV data), and are you combining the layers you actually need?",[16,424,425,426,431],{},"Designing and monitoring all of this yourself on a self-managed Proxmox + K3s environment is real, ongoing work. If you'd rather hand off storage design as part of your cluster operations, ",[64,427,430],{"href":428,"rel":429},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fkubo\u002Fon-premise",[68],"Kubo On-Premise"," is worth a look — it lets you build on your own infrastructure, Proxmox included, while getting fully managed Kubernetes operations that account for exactly this kind of storage design and monitoring.",[16,433,434,435,438],{},"And if you'd rather start by trying a managed K8s on the public cloud, ",[64,436,142],{"href":140,"rel":437},[68]," Cloud bakes in the same storage design know-how. It's built on lightweight K3s, and keeps monthly costs lower than EKS or AKS — a difference that matters even more once you factor in the operational design work this article has walked through.",[16,440,441,442,447],{},"Whether you keep fighting node-local storage traps yourself or hand the whole design over to a managed service is a real trade-off. If you're not sure which way to go, feel free to reach out via ",[64,443,446],{"href":444,"rel":445},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[68],"our contact page"," and talk it through.",{"title":449,"searchDepth":450,"depth":450,"links":451},"",2,[452,453,457,460,463,464],{"id":13,"depth":450,"text":14},{"id":52,"depth":450,"text":53,"children":454},[455],{"id":101,"depth":456,"text":102},3,{"id":146,"depth":450,"text":147,"children":458},[459],{"id":193,"depth":456,"text":194},{"id":226,"depth":450,"text":227,"children":461},[462],{"id":287,"depth":456,"text":288},{"id":306,"depth":450,"text":307},{"id":376,"depth":450,"text":377},"2026-09-12","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending\u002Ftitle.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending",{"title":5,"description":466},"blog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending",[476,477,478,479,480,481,482],"k3s","kubernetes","storage","local-path-provisioner","persistent-volume","node-affinity","zfs","-xLekODJ-ssd31qnKRziLtWG8W9RmbkHfQfLPhn1uuE",[485,494,503,512,522,532],{"path":486,"title":487,"description":488,"date":489,"tags":490},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[476,477,491,492,493],"hpa","keda","autoscaling",{"path":495,"title":496,"description":497,"date":498,"tags":499},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[476,477,500,337,501,502],"proxmox","live-migration","high-availability",{"path":504,"title":505,"description":506,"date":507,"tags":508},"\u002Fblog\u002Fen\u002Fk3s-harbor-private-registry-docker-hub-rate-limit","Our CI Died at Midnight from Docker Hub's Pull Rate Limit. Designing a Harbor Private Registry for K3s\u002FKubo","Ever had a K3s cluster blocked by Docker Hub's pull rate limit? Here's how a Harbor private registry with Proxy Cache dodges the limit, plus vulnerability scanning, signing, and production-grade operational design.","2026-09-08",[476,477,509,510,511],"harbor","container-registry","private-registry",{"path":513,"title":514,"description":515,"date":516,"tags":517},"\u002Fblog\u002Fen\u002Faks-cost-optimization-fixed-cost-structure-k3s-comparison","You Optimized AKS Costs to the Letter. So Why Won't the Bill Move?","Right-sizing nodes and adopting Spot VMs on AKS often plateaus because LoadBalancer-per-service billing and monitoring add-ons are structural fixed costs that operational tuning can't touch. We break down the real bottleneck and compare it to K3s's lightweight cost structure.","2026-09-07",[476,477,518,519,520,521],"aks","azure","cost-optimization","managed-kubernetes",{"path":523,"title":524,"description":525,"date":526,"tags":527},"\u002Fblog\u002Fen\u002Fk3s-ai-agent-opentelemetry-observability","Your AI Agent Is Running on K3s, But You Have No Idea What It's Doing. Tracing Non-Deterministic Execution with OpenTelemetry","MCP servers and AI agents on K3s don't reveal why they called a given tool just from logs. Here's how OpenTelemetry's GenAI semantic conventions make non-deterministic execution observable.","2026-09-06",[476,477,528,529,530,531],"opentelemetry","distributed-tracing","genai","telemetry",{"path":533,"title":534,"description":535,"date":536,"tags":537},"\u002Fblog\u002Fen\u002Fk3s-zero-trust-kubectl-management-access","Stop Trusting Your Corporate VPN: 3 Steps to Make K3s Management Access Boundary-Free","Are you still designing K3s cluster management access around a corporate VPN? Learn 3 practical steps to secure kubectl and Rancher dashboard access with identity instead of network perimeter, plus the pitfall most teams hit when they implement zero trust.","2026-09-05",[476,477,538,539,540],"zero-trust","security","rancher",1792488904023]