[{"data":1,"prerenderedAt":431},["ShallowReactive",2],{"blog-en-k3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral":3,"blog-related-en-k3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral":376,"blog-en-k3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral-alt":364},{"id":4,"title":5,"author":6,"body":7,"date":358,"description":359,"extension":360,"image":361,"locale":362,"meta":363,"navigation":364,"path":365,"seo":366,"stem":367,"tags":368,"__hash__":375},"blog\u002Fblog\u002Fen\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral.md","Your HPA Config Is Correct. So Why Won't Pods Scale? The Real Cause of 'Unknown Metrics' in K3s and the HPA\u002FVPA Death Spiral","Kubo Team",{"type":8,"value":9,"toc":347},"minimark",[10,15,29,36,59,63,80,93,99,103,116,123,129,138,142,145,156,162,170,174,187,253,259,263,266,310,316,323,327,333],[11,12,14],"h2",{"id":13},"the-hpa-yaml-is-correct-not-a-single-pod-gets-added","The HPA YAML Is Correct. Not a Single Pod Gets Added",[16,17,18,19,23,24,28],"p",{},"You configured a HorizontalPodAutoscaler (HPA) on your K3s cluster. ",[20,21,22],"code",{},"kubectl apply"," succeeded. The CPU utilization threshold and the min\u002Fmax replica counts are correct no matter how many times you re-read the manifest. Yet when load spikes, not a single Pod gets added. When engineers say ",[25,26,27],"strong",{},"\"HPA isn't scaling on K3s,\""," the root cause is almost never a bug in HPA itself — 9 times out of 10 it's hiding one layer beneath it, in resource configuration or the metrics pipeline.",[16,30,31],{},[32,33],"img",{"alt":34,"src":35},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection01.webp",[16,37,38,39,42,43,46,47,50,51,58],{},"If running ",[20,40,41],{},"kubectl get hpa"," shows the ",[20,44,45],{},"TARGETS"," column stuck at ",[20,48,49],{},"\u003Cunknown>\u002F50%",", that's not a sign that HPA is broken — it's a sign that HPA isn't receiving the information it needs. This article walks through the three root causes that show up repeatedly in K3s operations, a diagnostic process that isolates the failing layer in about three minutes, and a design approach that prevents recurrence. Whether you have an observability stack that visualizes autoscaling behavior (",[52,53,57],"a",{"href":54,"rel":55},"https:\u002F\u002Fkubo.hexabase.io\u002F",[56],"nofollow","Kubo"," ships with Prometheus + Grafana built in) makes a real difference in how fast you can pin down the cause.",[11,60,62],{"id":61},"cause-1-missing-resource-requests-a-silent-disablement","Cause 1: Missing Resource Requests — a Silent Disablement",[16,64,65,66,69,70,75,76,79],{},"The most commonly overlooked cause is a missing ",[20,67,68],{},"resources.requests"," on the container. ",[52,71,74],{"href":72,"rel":73},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Frun-application\u002Fhorizontal-pod-autoscale-walkthrough\u002F",[56],"Kubernetes' official HorizontalPodAutoscaler walkthrough"," explicitly sets ",[20,77,78],{},"requests.cpu"," on the target Deployment in its example, because HPA uses that value as the denominator when calculating utilization.",[16,81,82,83,88,89,92],{},"What makes this tricky is that the HPA controller doesn't throw an error when requests are missing. One technical writeup notes that \"HPA assumes resource requests are defined, and without them it cannot compute a utilization percentage\" (",[52,84,87],{"href":85,"rel":86},"https:\u002F\u002Foneuptime.com\u002Fblog\u002Fpost\u002F2026-02-09-hpa-not-scaling-missing-metrics\u002Fview",[56],"per oneuptime.com's explanation","). In other words, this never surfaces as an obvious misconfiguration — ",[25,90,91],{},"the affected metric is silently skipped, and HPA just keeps running in a state where it believes there's nothing to do",". You think you're monitoring the workload, but in reality nothing is being measured at all. This silent disablement is the single most common reason HPA fails to fire in K3s production environments.",[16,94,95],{},[32,96],{"alt":97,"src":98},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection02.webp",[11,100,102],{"id":101},"cause-2-metrics-pipeline-latency-kubelet-metrics-server-hpa","Cause 2: Metrics Pipeline Latency (kubelet → metrics-server → HPA)",[16,104,105,106,111,112,115],{},"Even when requests are configured correctly, you're not out of the woods yet. The utilization data HPA reads only becomes available after passing through a chain: kubelet → Metrics Server → the HPA controller → the scheduler → Pod startup → the readiness probe. According to the ",[52,107,110],{"href":108,"rel":109},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fmetrics-server\u002Fblob\u002Fmaster\u002FFAQ.md",[56],"metrics-server FAQ on kubernetes-sigs",", the default scrape interval for metrics-server is ",[25,113,114],{},"60 seconds",", and setting it below 15 seconds is not recommended, since that's the resolution at which kubelet itself calculates metrics.",[16,117,118,119,122],{},"On top of that collection interval, the HPA controller's evaluation cycle, Pod scheduling, container startup, and the time to pass the readiness probe all add up — ",[25,120,121],{},"it's a structural given that anywhere from tens of seconds to a few minutes will elapse between a traffic spike and the moment scaling actually completes",". If HPA \"feels slow,\" in most cases that isn't a malfunction; it's simply the physical characteristic of this pipeline, and alert thresholds and load-test scenarios should be designed around that latency from the outset.",[16,124,125],{},[32,126],{"alt":127,"src":128},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection03.webp",[16,130,131,132,137],{},"The same pipeline is used even on small K3s clusters, so if the configuration requirements documented in the ",[52,133,136],{"href":134,"rel":135},"https:\u002F\u002Fgithub.com\u002Fkubernetes-sigs\u002Fmetrics-server",[56],"metrics-server GitHub repository"," — such as enabling the aggregation layer and configuring kubelet certificate validation — aren't met, metrics can't even be collected at the front of the pipeline. Because K3s is a lightweight distribution, some components are sometimes omitted from the default setup, so confirming a healthy metrics-server deployment should be your first diagnostic step.",[11,139,141],{"id":140},"cause-3-the-hpavpa-death-spiral","Cause 3: The HPA\u002FVPA \"Death Spiral\"",[16,143,144],{},"Requests are configured, and metrics are flowing correctly. Yet scaling behavior is still unstable — Pod counts oscillate, or the workload scales out far more aggressively than expected. If that's what you're seeing, suspect a VerticalPodAutoscaler (VPA) running alongside HPA.",[16,146,147,152,153],{},[52,148,151],{"href":149,"rel":150},"https:\u002F\u002Fscaleops.com\u002Fblog\u002Fhpas-three-architectural-flaws-and-why-your-autoscaling-keeps-failing\u002F",[56],"ScaleOps' analysis"," describes the self-reinforcing loop that occurs when HPA and VPA target the same metric. When HPA scales out horizontally, per-Pod load drops, and VPA interprets this as evidence that requests can be lowered. When VPA lowers requests, the denominator in HPA's utilization calculation (the request value) shrinks, so the same absolute usage now looks like a spike in utilization, prompting HPA to scale out even further. This repeating cycle is what's known as the ",[25,154,155],{},"\"HPA\u002FVPA death spiral.\"",[16,157,158],{},[32,159],{"alt":160,"src":161},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection04.webp",[16,163,164,169],{},[52,165,168],{"href":166,"rel":167},"https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fautoscaler\u002Ftree\u002Fmaster\u002Fvertical-pod-autoscaler",[56],"VPA's CRD design"," is built around automatically adjusting requests, and this structural conflict surfaces whenever both controllers run in Auto mode on the same workload without a clear division of responsibility between them.",[11,171,173],{"id":172},"diagnostic-steps-isolating-the-bottleneck-layer-in-three-minutes","Diagnostic Steps: Isolating the Bottleneck Layer in Three Minutes",[16,175,176,177,182,183,186],{},"Since the root cause can live in one of three distinct layers, systematically narrowing it down is faster than blindly re-reading YAML. The ",[52,178,181],{"href":179,"rel":180},"https:\u002F\u002Fwww.codewithkarani.com\u002Fblog\u002Fkubernetes-hpa-not-scaling-describe-conditions",[56],"diagnostic guide from codewithkarani",", which walks through checking the Conditions from ",[20,184,185],{},"kubectl describe hpa",", is a solid starting flow.",[188,189,190,212,226,239],"ol",{},[191,192,193,200,203,204,207,208,211],"li",{},[25,194,195,196,199],{},"Run ",[20,197,198],{},"kubectl describe hpa \u003Cname> -n \u003Cnamespace>"," and read the Conditions from top to bottom.",[20,201,202],{},"AbleToScale: False"," points to a problem with the scale target reference or RBAC permissions. ",[20,205,206],{},"ScalingActive: False"," lets you distinguish, via its message, between \"no metrics returned\" (a metrics-server-side problem) and \"missing request for cpu\" (unset requests). ",[20,209,210],{},"ScalingLimited: True"," simply means you've hit the min\u002Fmax boundary — it isn't a failure.",[191,213,214,225],{},[25,215,216,217,220,221,224],{},"Check whether metrics-server itself is returning values with ",[20,218,219],{},"kubectl top pods"," and ",[20,222,223],{},"kubectl top nodes",".","\nIf no values come back here, the problem is metrics-server's configuration, not HPA.",[191,227,228,234,235,238],{},[25,229,230,231,224],{},"Check the request configuration directly with ",[20,232,233],{},"kubectl get deployment \u003Cname> -o jsonpath='{.spec.template.spec.containers[0].resources}'","\nIf ",[20,236,237],{},"requests"," is empty, Cause 1 is confirmed.",[191,240,241,244,245,248,249,252],{},[25,242,243],{},"Check whether a VPA is applied to the same workload.","\nIf a VPA object's ",[20,246,247],{},"updateMode"," is ",[20,250,251],{},"Auto"," and it's watching the same metric as HPA, suspect the Cause 3 death spiral.",[16,254,255],{},[32,256],{"alt":257,"src":258},"section05","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection05.webp",[11,260,262],{"id":261},"design-fixes-setting-requests-tuning-stabilizationwindow-and-when-to-move-to-keda","Design Fixes: Setting Requests, Tuning stabilizationWindow, and When to Move to KEDA",[16,264,265],{},"Once diagnosis identifies the cause, translate it into a design that prevents recurrence.",[267,268,269,278,292,298],"ul",{},[191,270,271,274,275,277],{},[25,272,273],{},"Set requests based on measured usage."," Don't use a low placeholder value — set ",[20,276,237],{}," based on actual measured CPU\u002Fmemory consumption under normal load.",[191,279,280,283,284,287,288,291],{},[25,281,282],{},"Don't let HPA and VPA share the same metric."," If HPA is driven by CPU, target VPA at memory only, or set ",[20,285,286],{},"updateMode: \"Off\""," or ",[20,289,290],{},"\"Initial\""," so it stops adjusting automatically and only surfaces recommendations for reference.",[191,293,294,297],{},[25,295,296],{},"Tune the scale-down stabilizationWindow."," To prevent flapping (rapid scale-up\u002Fscale-down cycles), configure a sufficiently long stabilization window on the scale-down side so Pods aren't removed the instant load dips.",[191,299,300,303,304,309],{},[25,301,302],{},"Consider moving to KEDA if queue length or event counts are your leading indicator."," For workloads where CPU\u002Fmemory alone can't capture the early signal of incoming load (such as message queue processing), the event-driven scaling provided by ",[52,305,308],{"href":306,"rel":307},"https:\u002F\u002Fkeda.sh\u002F",[56],"KEDA"," is a better fit. KEDA doesn't replace HPA — it's designed to sit above it and extend the set of scaling triggers while coexisting with it.",[16,311,312],{},[32,313],{"alt":314,"src":315},"section06","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Fsection06.webp",[16,317,318,319,322],{},"All of these design changes assume one thing: that metrics are actually visible and correct. Writing the configuration isn't enough to be confident it works — you can only verify the design is functioning once you can actually see request values, utilization, and scaling history in Prometheus or Grafana. ",[52,320,57],{"href":54,"rel":321},[56]," ships with Prometheus + Grafana built in, so you can immediately check metrics-server latency and HPA\u002FVPA scaling behavior on a dashboard.",[11,324,326],{"id":325},"conclusion-autoscaling-isnt-set-and-forget-its-set-and-observe","Conclusion: Autoscaling Isn't \"Set and Forget\" — It's \"Set and Observe\"",[16,328,329,330,332],{},"When HPA won't scale on K3s, the cause is rarely a defect in HPA itself. It usually lives in one of three layers: unset resource requests, a metrics-server configuration gap or collection-interval delay, or a design conflict between HPA and VPA. Starting from the Conditions in ",[20,331,185],{}," to isolate the failing layer lets you skip the blind trial-and-error.",[16,334,335,336,340,341,346],{},"The other key point is that autoscaling isn't something you configure once and walk away from — it only delivers on its promise when you keep observing it continuously. If building an observability stack from scratch feels like too much overhead, it's worth considering a managed Kubernetes offering like ",[52,337,339],{"href":54,"rel":338},[56],"Kubo Cloud",", which is K3s-based and ships with Prometheus + Grafana out of the box. It costs less than AWS EKS or Azure AKS while staying compatible with standard Kubernetes, letting you run the diagnostic workflow described in this article entirely from a dashboard. If you're not confident in your current autoscaling design, ",[52,342,345],{"href":343,"rel":344},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[56],"get in touch"," and we can help you work through it.",{"title":348,"searchDepth":349,"depth":349,"links":350},"",2,[351,352,353,354,355,356,357],{"id":13,"depth":349,"text":14},{"id":61,"depth":349,"text":62},{"id":101,"depth":349,"text":102},{"id":140,"depth":349,"text":141},{"id":172,"depth":349,"text":173},{"id":261,"depth":349,"text":262},{"id":325,"depth":349,"text":326},"2026-09-02","You set up an HPA in K3s and Pods still won't scale. The usual culprits are unset resource requests silently disabling metrics, and a self-reinforcing HPA\u002FVPA feedback loop. Here's how to diagnose and fix it.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral\u002Feyecatch.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral",{"title":5,"description":359},"blog\u002Fen\u002Fk3s-hpa-not-scaling-metrics-unknown-vpa-death-spiral",[369,370,371,372,373,374],"k3s","kubernetes","hpa","vpa","autoscaling","metrics-server","iZ5n1AGHBJxiJDvgYZWMgzzzlnv8pSPoiyEBrqrYLbE",[377,384,392,401,410,421],{"path":378,"title":379,"description":380,"date":381,"tags":382},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[369,370,371,383,373],"keda",{"path":385,"title":386,"description":387,"date":388,"tags":389},"\u002Fblog\u002Fen\u002Fkubernetes-hpa-vpa-autoscaling","The Complete Kubernetes Autoscaling Guide: HPA, VPA, and KEDA in Practice","Master Kubernetes autoscaling with HPA, VPA, and KEDA. Learn each tool's strengths, combination patterns, and cost optimization impact with practical examples.","2026-05-27",[370,373,371,372,383,390,391],"karpenter","cost-optimization",{"path":393,"title":394,"description":395,"date":396,"tags":397},"\u002Fblog\u002Fen\u002Fkubernetes-cpu-throttling-connection-pool-exhaustion","The Trap of 'CPU Usage Is Low, So We're Fine': The Hidden Ceiling of Throttling and Connection Pool Exhaustion Kubernetes Won't Show You","When adding more Pods to Kubernetes doesn't fix latency, the real cause may not be CPU at all, but invisible throttling or database connection pool exhaustion. Here's how to diagnose and fix the hidden ceiling.","2026-07-28",[370,369,398,399,400,371],"performance-tuning","cpu-throttling","connection-pooling",{"path":402,"title":403,"description":404,"date":405,"tags":406},"\u002Fblog\u002Fen\u002Fai-manifest-generation-kubernetes-architecture-bottleneck","AI Can Write a YAML File in One Second, But Your Cluster Won't Get Any Faster. Why the Real Bottleneck in Kubernetes Operations Is Architecture, Not Code","Generating Kubernetes manifests at AI speed doesn't make production Kubernetes operations faster. This article breaks down three real bottlenecks — CPU throttling, the HPA\u002FVPA conflict, and database connection starvation — and how to split the work between AI and humans.","2026-07-18",[370,369,407,373,408,409],"resource-management","ai-ops","devops",{"path":411,"title":412,"description":413,"date":414,"tags":415},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[369,370,416,417,418,419,420],"storage","local-path-provisioner","persistent-volume","node-affinity","zfs",{"path":422,"title":423,"description":424,"date":425,"tags":426},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[369,370,427,428,429,430],"proxmox","etcd","live-migration","high-availability",1792488903451]