Factory floors, data center racks, the back room of a retail store — as edge Kubernetes deployments spread into these environments, one problem always surfaces sooner or later: designing a K3s load balancer around BGP, a concern you never had to think about in the cloud. On EKS or AKS, writing type: LoadBalancer is enough to get an external IP automatically. But on bare metal, where no cloud API exists, you have to build that mechanism yourself.
How Does "Service type: LoadBalancer" Work Without a Cloud LB?
According to the Kubernetes documentation, a type: LoadBalancer Service normally goes through the Cloud Controller Manager, which calls the cloud provider's load balancer API to automatically assign an external IP. This mechanism is tightly coupled to public clouds like AWS, Azure, and GCP.
In bare-metal environments — factories, branch sites, edge locations — there is no cloud API, so this automatic assignment simply doesn't happen. This is where software load balancers like MetalLB come in: they use standard routing protocols to "announce" an external IP onto the network. The MetalLB documentation describes two operating modes: Layer2 (ARP/NDP) and BGP.
If you're running K3s, there's one more thing to watch for. K3s ships with a built-in load balancer called ServiceLB (formerly klipper-lb), enabled by default. If you want to install MetalLB or another implementation instead, the K3s documentation requires you to pass --disable servicelb on every server node — otherwise the two implementations will fight over which one owns the external IP assignment.
If you'd rather not own this design decision from scratch, a K3s-based managed service like Kubo is another option — it removes the need to build the network design yourself.
Where Layer2 Mode Hits a Wall: A Single Node, ARP Caches, and Bandwidth
MetalLB's Layer2 mode, according to its official documentation, works through leader election: "one node takes responsibility for announcing a service IP to the local network." ARP (IPv4) or NDP (IPv6) responses then make it look as if a single machine holds multiple external IP addresses.
That's fine in a test environment. But in production on-prem or edge deployments, two structural limits appear:
- Single-node bottleneck: All traffic to a given external IP funnels into whichever node was elected leader. No matter how many nodes you add to the cluster, that one service's inbound bandwidth can never exceed the physical bandwidth of a single node.
- Failover delay: When the leader node goes down, a new node has to resend ARP responses to tell clients the target has switched. In most environments this converges within seconds, but clients holding stale ARP cache entries can experience delays measured in minutes.
The Red Hat engineering blog makes the same point explicitly: Layer2 mode turns a node into "a single point of failure that limits performance." As a K3s cluster grows from three nodes to ten in an edge rollout, this constraint becomes a hard ceiling on that service's scalability.
Moving to BGP: MetalLB's BGP Mode vs. Cilium's BGP Control Plane
The standard way past Layer2's limits is BGP mode. According to MetalLB's BGP concept guide, in BGP mode every node in the cluster establishes a BGP peering session with the upstream router, and the same external IP is advertised simultaneously from multiple nodes. If the router supports ECMP (Equal-Cost Multi-Path), traffic is actually load-balanced across those nodes based on packet hashing.
The Red Hat blog sums up BGP mode's advantages as stateless load balancing, simultaneous advertisement from multiple nodes, and fast failure detection via the BFD protocol. MetalLB supports delegating the actual BGP session handling to an FRR-based backend called FRR-K8s, which unlocks capabilities Layer2 mode doesn't have, such as combining BFD sessions with IPv6 advertisement.
There's a second option if your CNI is Cilium: Cilium's BGP Control Plane. According to the Cilium documentation, the BGP Control Plane advertises Pod network and Service external IPs over BGP, while LB-IPAM handles IP address allocation. If your cluster already runs Cilium as its CNI, you can consolidate BGP load balancing entirely into the CNI layer without adding MetalLB as a separate component.
# Example: simplified Cilium BGP Control Plane configuration
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
name: rack01-bgp-policy
spec:
nodeSelector:
matchLabels:
rack: "01"
virtualRouters:
- localASN: 65001
neighbors:
- peerAddress: "10.0.0.1/32"
peerASN: 65000
serviceSelector:
matchLabels:
advertise: "bgp"
Note that MetalLB itself used to offer an integrated mode called MetalLB BGP ControlPlane that paired with Cilium's data plane, but that mode is now in a maintenance-only phase receiving security fixes, with migration toward Cilium's newer BGP Control Plane recommended instead. Which path makes sense depends heavily on your existing CNI choice — that's the real starting point for a migration decision.
If you want BGP handled entirely by dedicated switches, alternative OSS implementations like Porter exist too, but in terms of track record and documentation depth within K3s environments, MetalLB and Cilium remain the two realistic starting points.
For teams that would rather not own this migration end-to-end, Kubo On-Premise is an option. It's K3s-based, supports air-gapped environments, and lets an operations team hand off network design — including BGP peering — without vendor lock-in. The tighter your external network connectivity requirements (think financial institutions or manufacturing edge sites), the bigger the payoff.
Three Things to Check Before You Migrate
Moving from Layer2 mode to BGP mode isn't something you can finish purely by configuring the Kubernetes cluster. In practice, there are three things you always need to confirm.
1. Have you disabled K3s's built-in ServiceLB (klipper)?
As mentioned above, K3s enables ServiceLB by default. Per MetalLB's K3s configuration guide, installing MetalLB without disabling it first causes the two implementations to compete over external IP assignment, leading to unpredictable load balancer behavior. Set --disable servicelb on every server node first, then install MetalLB or enable Cilium's BGP features — the order matters.
2. Do your upstream routers/switches support BGP, and have you aligned with the network team?
BGP configuration doesn't stop at the cluster boundary. You also need BGP peer settings (ASN, peer IP addresses, route filters) on the upstream router or switch side, and in most organizations that falls under the network team's authority. This alignment process — unnecessary under Layer2 mode — tends to be the biggest practical hurdle in a BGP migration.
3. Have you verified failover behavior?
Whether ECMP routes get withdrawn correctly on node failure, and whether BFD-based fast detection actually works, are things you must verify before going to production. As the Red Hat blog notes, combining BFD can shorten failure detection to well under a second — but that depends entirely on router-side support, which is why pre-verification is non-negotiable.
Summary
For bare-metal and edge K3s clusters that can't rely on a cloud load balancer, Layer2 (ARP) mode load balancing is really only a compromise fit for test environments. It inevitably runs into structural limits — traffic concentrating on a single node, delayed failover — which is why migrating to BGP mode becomes practically inevitable for production use.
Which path to take depends on your CNI setup. If you're already running Cilium, consolidating into its BGP Control Plane and LB-IPAM is the sensible choice; for any other CNI, MetalLB's BGP mode is the realistic option. Either way, the deciding factor in a successful migration is doing the unglamorous groundwork: disabling K3s's ServiceLB and reaching agreement with the network team.
As Kubernetes spreads beyond the cloud into factories, retail stores, and even vehicles, this kind of network design decision becomes unavoidable. K3s-based managed services like Kubo and Kubo On-Premise can take on this design and operational burden for you. If you'd rather not shoulder the verification cost of a BGP migration yourselves, reaching out is worth considering.