1. "Ping works, but curl just hangs" — the mystery slowdown in K3s clusters
Have you ever run into this while operating a K3s cluster? A ping between Pods returns instantly, but an HTTP request between the same two Pods establishes a connection and then just hangs, never returning a response. DNS resolution sometimes works fine, while only larger records time out.
Because a restart sometimes "fixes" it, many teams write it off as a fluke. But this is a classic symptom of an MTU mismatch in the K3s overlay network. The same class of problem is well known in Docker's overlay networking too — any environment where container-to-container traffic passes through an encapsulation technology like VXLAN can hit this, not just K3s or Kubernetes. This article walks through why the symptom occurs, how to diagnose it, and how to fix it for good.
The root cause is that the "effective MTU" between Pods ends up smaller than what the cluster was designed for. K3s's default CNI, Flannel, encapsulates Pod-to-Pod traffic using a VXLAN backend (K3s official documentation). This encapsulation adds overhead, and under the wrong conditions it produces an especially nasty failure mode: small packets get through, but large ones silently disappear.
This kind of networking pitfall is also part of the "invisible cost" of designing and operating a K3s cluster from scratch. In fact, when evaluating a managed K3s service like Kubo, it's worth adding "how much of this is already handled for you" as one of your comparison criteria.
2. The culprit: VXLAN encapsulation overhead — how 1500 MTU gets whittled down
Why do we need an overlay network at all?
Kubernetes is designed around the assumption that Pods on different nodes each have their own unique IP address and can communicate directly, without NAT. To meet this requirement, Kubernetes builds its Pod network using plugins that conform to the CNI (Container Network Interface) specification (Kubernetes official documentation). In environments where the physical network can't natively route the Pod IP address space, an overlay technology like VXLAN encapsulates Pod-to-Pod traffic, creating a "virtual direct connection" layered on top of the existing network path.
Where the 50 bytes of VXLAN overhead come from
According to RFC 7348, the specification that defines VXLAN, VXLAN encapsulates the original Pod Ethernet frame inside four additional layers: an outer Ethernet header, an outer IP header, a UDP header, and a VXLAN header. Over IPv4, the typical overhead breaks down as follows.
- Outer Ethernet header: 14 bytes
- Outer IPv4 header: 20 bytes
- Outer UDP header: 8 bytes
- VXLAN header: 8 bytes
- Total: 50 bytes
In other words, even if the physical network's standard MTU is 1500 bytes, the effective MTU for VXLAN-encapsulated Pod-to-Pod traffic shrinks to 1450 bytes. Calico's own MTU configuration documentation makes the same point explicitly: when using VXLAN (IPv4), you should set the MTU to the physical network's MTU minus the encapsulation overhead — 1450 for IPv4 VXLAN, or 1480 for IP-in-IP, which has a 20-byte overhead (Calico official documentation).
The trouble starts when this "1450" value isn't consistent across the entire cluster. Whether it's the VPC's MTU in the cloud, a switch's MTU on-prem, or a VPN tunnel's MTU for inter-node traffic over VPN — if this calculation is off anywhere along the path, packets stop fitting into the expected size.
3. Why doesn't Path MTU Discovery save you? The truth about PMTU blackholes
How PMTUD is supposed to work
TCP/IP includes a mechanism called Path MTU Discovery (PMTUD), which automatically detects the largest packet size that can actually pass between a source and destination. When an IPv4 packet with the "Don't Fragment" (DF) flag set exceeds the MTU somewhere along the path, the router at that point drops the packet and sends an ICMP "Fragmentation Needed" (Type 3, Code 4) message back to the sender. The sender receives this ICMP message, shrinks its MTU accordingly, and retransmits (AWS official documentation, Path MTU Discovery – Wikipedia).
When ICMP gets blocked, you get a "blackhole"
The problem is that many cloud environments, overlay networks, and security groups block this ICMP message outright. When that happens, the sender never learns that its MTU is too large, keeps sending packets of the same size, and those packets get silently dropped somewhere along the path, over and over. This condition is commonly known as a "PMTU blackhole."
Small packets — like ICMP echo requests used by ping, or the TCP three-way handshake itself — pass through without issue, while only packets with a larger payload, like HTTP responses, vanish. That's exactly what produces the seemingly contradictory symptom: "ping works, but HTTP just hangs." The more a K3s cluster's traffic passes through cloud VPCs, on-prem firewalls, or VPN appliances, the more likely this problem becomes.
4. Diagnosing it in practice — three checkpoints for pinpointing an MTU mismatch
Once you have a hypothesis, it's time to verify it on the actual cluster. Flannel's own troubleshooting guide walks through a three-stage approach for isolating MTU-related issues: checking the node's interface, checking the configuration file, and checking the container side (Flannel official troubleshooting guide). Here, we translate that three-stage approach into concrete commands you can run in practice.
Step 1: Check the MTU of the node's network interface
ip link show <interface used by Flannel>
Check whether the MTU of the interface Flannel is actually using — whether that's a physical NIC, a VPN interface, or an overlay interface — matches what you expect.
Step 2: Cross-check against the value Flannel wrote to subnet.env
cat /run/flannel/subnet.env
Flannel writes its MTU setting to this file on every node. Since the CNI plugin reads this value when configuring the Pod's network, check whether the value recorded here matches the expected figure (physical MTU minus 50 for VXLAN).
Step 3: Check the MTU of the virtual interface inside the Pod
kubectl exec <pod-name> -- ip link show eth0
Check the MTU set on the Pod's internal eth0 (veth) interface. If this value doesn't match what you found in Steps 1 and 2, suspect either a CNI configuration that didn't propagate correctly or an inconsistency between nodes.
If none of these checks turn up anything unusual, the problem likely isn't inside the cluster at all — look instead at other segments along the path, such as your cloud provider's VPC settings or an on-prem switch or router.
5. The permanent fix: MSS clamping and cloud-specific MTU design
Using TCP MSS clamping to sidestep fragmentation entirely
Since the root cause of a PMTU blackhole is blocked ICMP traffic, the ideal fix is to allow ICMP Type 3 Code 4 through every firewall along the path. But you can't always control every network device on that path. The practical workaround in that case is "TCP MSS clamping."
MSS clamping rewrites the MSS (Maximum Segment Size) value exchanged during the TCP three-way handshake, shrinking it to match the effective MTU of the path. This means TCP sends smaller segments from the very start, sidestepping the fragmentation problem entirely without relying on PMTUD. A typical implementation applies an iptables --clamp-mss-to-pmtu rule to the overlay network interface.
Cloud-specific MTU design values
Default MTU values for physical and virtual networks differ across cloud environments. It's worth designing your Pod-side MTU in advance by subtracting the VXLAN overhead (50 bytes) from whatever the underlying network MTU is.
| Environment | Network MTU | Recommended Pod MTU with VXLAN |
|---|---|---|
| Standard Ethernet | 1500 | 1450 |
| AWS (with jumbo frames enabled) | 9001 (AWS official documentation) | 8951 |
| GCP VPC (default) | 1460 (GCP official documentation) | 1410 |
Note that on AWS, traffic routed through an internet gateway is capped at 1500 bytes, so jumbo frames may not be usable for all inter-node paths depending on your routing (AWS official documentation). Redesigning your MTU with these cloud-specific constraints in mind is unglamorous work, but it's an important step in preventing production incidents.
Re-verifying this kind of fine-grained network design from scratch every time you build a cluster is a real, recurring operational cost. Kubo is a managed Kubernetes service built on K3s that ships with its network layer — including Flannel/Calico MTU settings — already validated. Compared to building EKS or AKS yourself, Kubo covers these operational details starting at ¥48,000/month.
6. Summary
An overlay network is one of those parts of a Kubernetes cluster that you normally never have to think about — it's "wiring you don't notice." But when VXLAN's encapsulation overhead combines with a PMTU blackhole, the result is a notoriously hard-to-diagnose failure: ping works, but HTTP just hangs.
The three-step diagnostic process covered here — checking the node NIC's MTU, cross-checking it against subnet.env, and checking the Pod's internal interface — gives you a systematic way to isolate what otherwise looks like an unexplainable network slowdown. And MSS clamping and cloud-specific MTU design aren't one-and-done settings; they're items worth re-verifying every time you stand up a new cluster.
You don't have to keep wrestling with the cost, complexity, and vendor lock-in of EKS or AKS. With K3s-based Kubo, you get a fully validated, production-grade Kubernetes environment — including this kind of network design — at a fraction of the cost. If your infrastructure team is tired of losing time to MTU-related incidents, get in touch for a conversation.