[{"data":1,"prerenderedAt":290},["ShallowReactive",2],{"blog-en-k3s-ai-inference-gpu-scheduling-dra-keda-kserve":3,"blog-related-en-k3s-ai-inference-gpu-scheduling-dra-keda-kserve":235,"blog-en-k3s-ai-inference-gpu-scheduling-dra-keda-kserve-alt":223},{"id":4,"title":5,"author":6,"body":7,"date":217,"description":218,"extension":219,"image":220,"locale":221,"meta":222,"navigation":223,"path":224,"seo":225,"stem":226,"tags":227,"__hash__":234},"blog\u002Fblog\u002Fen\u002Fk3s-ai-inference-gpu-scheduling-dra-keda-kserve.md","Kubernetes Hasn't Changed Its Design in 10 Years — Here Are 3 Assumptions Your K3s Cluster Must Rethink for AI Inference Workloads","Kubo Team",{"type":8,"value":9,"toc":207},"minimark",[10,15,19,30,34,42,51,60,68,72,81,90,105,112,116,119,128,137,150,154,167,176,183,187,190],[11,12,14],"h2",{"id":13},"_1-why-the-10-year-old-design-can-no-longer-carry-ai-workloads","1. Why the \"10-Year-Old Design\" Can No Longer Carry AI Workloads",[16,17,18],"p",{},"It has been nearly a decade since Kubernetes became widely adopted. Back then, the core challenge was deploying microservice-based, stateless web applications in a declarative, reproducible way — scaling pods up and down, load-balancing through Services, and autoscaling based on CPU and memory utilization. That design philosophy still works well for a huge number of clusters today.",[16,20,21,22,29],{},"But once AI agents and LLM inference workloads moved into production, the picture changed. Inference workloads are governed by constraints that traditional CPU\u002Fmemory-centric metrics simply can't see: GPU memory occupancy and request queue depth. Even teams running K3s-based clusters now need to rethink three assumptions — how GPUs are counted, how scaling decisions are made, and how serving is built — to reliably support AI workloads. ",[23,24,28],"a",{"href":25,"rel":26},"https:\u002F\u002Fkubo.hexabase.io\u002F",[27],"nofollow","Kubo",", a managed service, continues to update its K3s cluster design to reflect exactly this shift in assumptions.",[11,31,33],{"id":32},"_2-assumption-1-gpus-are-just-countable-integer-resources","2. Assumption #1 — \"GPUs Are Just Countable Integer Resources\"",[16,35,36,37,41],{},"Until recently, the standard way to handle GPUs in Kubernetes was to use the NVIDIA Device Plugin to count how many GPUs exist on each node, with pods requesting them as integers like ",[38,39,40],"code",{},"nvidia.com\u002Fgpu: 1",". This approach can't account for GPU model, generation, memory capacity, or topology.",[16,43,44,45,50],{},"Dynamic Resource Allocation (DRA), which reached General Availability in Kubernetes 1.34, resolves this constraint. According to the ",[23,46,49],{"href":47,"rel":48},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2025\u002F09\u002F01\u002Fkubernetes-v1-34-dra-updates\u002F",[27],"official Kubernetes blog",", DRA transforms how specialized hardware like GPUs and FPGAs is allocated: workloads declare the attributes they need, and the scheduler matches and assigns devices accordingly. The ResourceClaim and DeviceClass APIs have stabilized at v1 and are enabled by default.",[16,52,53,54,59],{},"NVIDIA's DRA driver makes this mechanism even more practical. According to the ",[23,55,58],{"href":56,"rel":57},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fdra-intro-install.html",[27],"official NVIDIA GPU Operator documentation",", GPUs can now be selected by attributes such as model, architecture, memory capacity, UUID, and PCI topology, and a single GPU can be safely shared across multiple workloads through MIG (Multi-Instance GPU) partitioning or time-slicing. Another key difference: while the traditional Device Plugin could only see kubelet-level availability on a node, DRA can monitor device health via DCGM.",[16,61,62,63,67],{},"This shift — from \"GPU as a countable integer\" to \"GPU as a declarative resource with attributes\" — changes the entire design of an AI inference cluster. Choosing a managed K3s platform like ",[23,64,66],{"href":25,"rel":65},[27],"Kubo Cloud"," reduces the burden of chasing this evolving resource model on your own from scratch.",[11,69,71],{"id":70},"_3-assumption-2-hpa-is-enough","3. Assumption #2 — \"HPA Is Enough\"",[16,73,74,75,80],{},"The standard Horizontal Pod Autoscaler (HPA) is designed to scale horizontally based on CPU and memory utilization. But in AI inference workloads, CPU utilization can look perfectly normal while the GPU-side queue overflows and response times degrade. A ",[23,76,79],{"href":77,"rel":78},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2026\u002F05\u002F27\u002Fgpu-autoscaling-on-kubernetes-with-keda-building-an-external-scaler\u002F",[27],"CNCF blog post"," describes this situation as the GPU actually doing the work becoming \"invisible to standard autoscaling,\" pointing out that this wastes accelerator capacity and worsens inference latency.",[16,82,83,84,89],{},"Kubernetes Event-driven Autoscaling (KEDA) addresses exactly this gap. As the ",[23,85,88],{"href":86,"rel":87},"https:\u002F\u002Fkeda.sh\u002Fdocs\u002F2.20\u002Fscalers\u002F",[27],"official KEDA documentation"," explains, KEDA scalers can activate or deactivate deployments and feed custom metrics based on external event sources like Kafka or Prometheus. Being able to trigger scaling on metrics HPA can't touch — inference queue length, GPU utilization — is what makes KEDA such a good fit for AI inference workloads.",[16,91,92,93,98,99,104],{},"As a real-world integration example, a Red Hat Developer article describes combining Kueue-based job queuing with KEDA's GPU-utilization-driven scale-to-zero, shrinking idle workloads and freeing GPUs for jobs waiting in the queue (",[23,94,97],{"href":95,"rel":96},"https:\u002F\u002Fdevelopers.redhat.com\u002Farticles\u002F2025\u002F08\u002F26\u002Foptimize-gpu-utilization-kueue-and-keda",[27],"Red Hat Developer","). Microsoft's AKS engineering blog similarly describes using KEDA's event-driven scaling to spin up additional instances before inference requests start backing up (",[23,100,103],{"href":101,"rel":102},"https:\u002F\u002Fblog.aks.azure.com\u002F2026\u002F02\u002F03\u002Fautoscale-inference-workloads-with-kaito",[27],"AKS Engineering Blog",").",[16,106,107,108,111],{},"This kind of mechanism fundamentally solves the problem of GPUs running idle overnight or on weekends and burning money for nothing. ",[23,109,66],{"href":25,"rel":110},[27]," ships with Prometheus + Grafana as standard, so you don't need to build the metrics foundation — GPU utilization, inference queue depth — from scratch.",[11,113,115],{"id":114},"_4-assumption-3-a-homegrown-http-server-is-enough-for-inference-serving","4. Assumption #3 — \"A Homegrown HTTP Server Is Enough for Inference Serving\"",[16,117,118],{},"Simply wrapping LLM inference in a homegrown HTTP server built with Flask or FastAPI quickly hits a wall once you need production-grade features: model switching, canary deployments, and traffic splitting across multiple models.",[16,120,121,122,127],{},"At the inference engine layer, vLLM has become the de facto standard. According to the ",[23,123,126],{"href":124,"rel":125},"https:\u002F\u002Fvllm.ai\u002Fblog\u002F2023-06-20-vllm",[27],"official vLLM blog",", the PagedAttention algorithm dramatically reduces KV cache memory fragmentation, delivering up to 24x higher throughput compared to HuggingFace Transformers. Inspired by virtual memory management, this technique is key to not wasting the scarce resource that is GPU memory.",[16,129,130,131,136],{},"On top of that sits the serving layer, and KServe is a leading choice here. According to the ",[23,132,135],{"href":133,"rel":134},"https:\u002F\u002Fkserve.github.io\u002Fwebsite\u002F",[27],"official KServe documentation",", KServe hides the complexity of autoscaling, networking, and health checks behind Kubernetes custom resources, using inference engines like vLLM as the backend while providing a standardized inference protocol and serverless-style scaling including scale-to-zero. Building on Kubernetes-native mechanisms instead of rolling your own control layer fundamentally changes the flexibility you have for adding models and controlling traffic.",[16,138,139,140,145,146,149],{},"If you're running AI agent or LLM inference infrastructure in production, you need to support this serving-layer design from the infrastructure up. Running an AI agent execution platform like ",[23,141,144],{"href":142,"rel":143},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fcaptain-ai\u002F",[27],"Captain.AI"," on top of ",[23,147,28],{"href":25,"rel":148},[27]," is one configuration well-suited to this approach.",[11,151,153],{"id":152},"_5-k3s-as-a-choice-combining-lightness-with-an-ai-ready-foundation","5. K3s as a Choice — Combining Lightness with an AI-Ready Foundation",[16,155,156,157,162,163,166],{},"K3s's reputation as a \"lightweight Kubernetes distribution\" often leads people to assume it's underpowered for AI inference workloads. In reality, K3s directly supports GPU workloads. According to the ",[23,158,161],{"href":159,"rel":160},"https:\u002F\u002Fdocs.k3s.io\u002Fadvanced",[27],"official K3s documentation",", installing the NVIDIA Container Runtime package lets K3s automatically detect and configure it as the containerd runtime — and if it isn't set as the default runtime, specifying ",[38,164,165],{},"runtimeClassName: nvidia"," in the pod spec is enough to use the GPU.",[16,168,169,170,175],{},"The NVIDIA GPU Operator documentation goes further, describing how the NRI (Node Resource Interface) plugin eliminates the need for manual environment variable configuration in environments like K3s and RKE2, where the containerd configuration path differs from the standard (",[23,171,174],{"href":172,"rel":173},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fcdi.html",[27],"NVIDIA GPU Operator documentation","). In other words, even the newest GPU resource models — DRA, MIG — work directly on top of a K3s cluster.",[16,177,178,179,182],{},"Being lightweight and meeting the infrastructure demands of the AI era are not in conflict. ",[23,180,28],{"href":25,"rel":181},[27]," is K3s-based while still offering a Rancher management layer, GitOps integration, and automated certificate management — delivering full-featured Kubernetes operations starting at ¥48,000\u002Fmonth, competitive with EKS and AKS. Running GPU-enabled workloads on this foundation means you don't need to build a separate infrastructure from scratch just for AI inference.",[11,184,186],{"id":185},"_6-conclusion","6. Conclusion",[16,188,189],{},"In the ten years since Kubernetes emerged, what infrastructure needs to deliver has changed dramatically. And yet Kubernetes itself, as an abstraction layer, keeps surviving by changing shape. The reason: the community keeps updating the underlying assumptions themselves — GPU allocation through Dynamic Resource Allocation, scaling decisions through KEDA, model serving architecture through KServe and vLLM.",[16,191,192,193,196,197,200,201,206],{},"If you're operating a K3s cluster, a good place to start is auditing which of these assumptions your own cluster is still built on. If you're planning to bring AI agents into production, building a solid infrastructure foundation is the first step. ",[23,194,28],{"href":25,"rel":195},[27]," lets you build the foundation to support AI-era workloads quickly, and take it all the way through to running ",[23,198,144],{"href":142,"rel":199},[27]," on top of it. Start with a ",[23,202,205],{"href":203,"rel":204},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[27],"contact request"," to talk through what your cluster actually needs to rethink.",{"title":208,"searchDepth":209,"depth":209,"links":210},"",2,[211,212,213,214,215,216],{"id":13,"depth":209,"text":14},{"id":32,"depth":209,"text":33},{"id":70,"depth":209,"text":71},{"id":114,"depth":209,"text":115},{"id":152,"depth":209,"text":153},{"id":185,"depth":209,"text":186},"2026-08-31","Ten years after Kubernetes launched, AI agents and LLM inference have upended three of its core assumptions: how GPUs are allocated, how autoscaling decisions are made, and how model serving is built. Here's how to rethink your K3s cluster with Dynamic Resource Allocation, KEDA, and KServe.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-inference-gpu-scheduling-dra-keda-kserve\u002Feyecatch.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-ai-inference-gpu-scheduling-dra-keda-kserve",{"title":5,"description":218},"blog\u002Fen\u002Fk3s-ai-inference-gpu-scheduling-dra-keda-kserve",[228,229,230,231,232,233],"k3s","kubernetes","ai-workload","gpu-scheduling","dynamic-resource-allocation","mlops","WG7g_pjuFiJUVB1RjjLk2HltWFb5ToepfkmDTf2eGE8",[236,244,251,260,271,280],{"path":237,"title":238,"description":239,"date":240,"tags":241},"\u002Fblog\u002Fen\u002Fkubernetes-mlops-gpu-scheduling-talent-shortage","In the Age of AI-Written Code, Why Are Infrastructure Engineers Getting Raises? Inside the 'MLOps Talent Shortage' Fueled by the Corporate AI Adoption Rush","Generative AI has made it possible for almost anyone to write code, yet as more companies adopt AI, demand for MLOps talent who can reliably run GPUs and model serving on Kubernetes keeps rising. Here's why, and how a managed Kubernetes platform can help.","2026-07-30",[229,228,233,242,231,243],"ai-infrastructure","devops",{"path":245,"title":246,"description":247,"date":248,"tags":249},"\u002Fblog\u002Fen\u002Fmlops-kubernetes-devops-ai-skills-2026","\"DevOps Engineers Are Becoming Obsolete\" Is a Lie. 5 MLOps Skills Every Kubernetes Operator Must Master in the AI Era","As AI automates infrastructure, are DevOps engineers really becoming irrelevant? The reality is the opposite — demand for MLOps Kubernetes talent is surging. Here are the 5 skills you need in 2026.","2026-07-11",[229,233,243,228,250,242],"ci-cd",{"path":252,"title":253,"description":254,"date":255,"tags":256},"\u002Fblog\u002Fen\u002Fai-platform-kubernetes-convergence-2026","66% of AI-Driven Companies Choose Kubernetes — Why AI Infrastructure Converges on K8s","CNCF's 2026 survey shows 66% of GenAI organizations run on Kubernetes. From GPU scheduling to multi-tenancy and cost optimization — here's why K8s has become the de facto AI platform, and how managed K8s can cut your build time to zero.","2026-07-10",[229,257,258,233,228,259],"ai","platform-engineering","gpu",{"path":261,"title":262,"description":263,"date":264,"tags":265},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[228,229,266,267,268,269,270],"storage","local-path-provisioner","persistent-volume","node-affinity","zfs",{"path":272,"title":273,"description":274,"date":275,"tags":276},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[228,229,277,278,279],"hpa","keda","autoscaling",{"path":281,"title":282,"description":283,"date":284,"tags":285},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[228,229,286,287,288,289],"proxmox","etcd","live-migration","high-availability",1792488901447]