[{"data":1,"prerenderedAt":321},["ShallowReactive",2],{"blog-en-k3s-cert-manager-acme-renewal-pitfalls":3,"blog-related-en-k3s-cert-manager-acme-renewal-pitfalls":266,"blog-en-k3s-cert-manager-acme-renewal-pitfalls-alt":255},{"id":4,"title":5,"author":6,"body":7,"date":249,"description":250,"extension":251,"image":252,"locale":253,"meta":254,"navigation":255,"path":256,"seo":257,"stem":258,"tags":259,"__hash__":265},"blog\u002Fblog\u002Fen\u002Fk3s-cert-manager-acme-renewal-pitfalls.md","A Saturday Morning K3s Outage from an Expired TLS Certificate: Designing ACME Automation That Actually Escapes Manual Renewal","Kubo Team",{"type":8,"value":9,"toc":231},"minimark",[10,19,22,33,36,41,48,63,68,79,83,92,96,105,109,126,129,138,142,148,151,155,162,166,179,188,192,195,199,205,208,214,217,221,224],[11,12,13,14,18],"p",{},"Your phone buzzes with a Slack alert on a Saturday morning. The production site is returning 503s. The cause: an expired TLS certificate. And this cluster had ",[15,16,17],"code",{},"cert-manager"," installed months ago. \"Wasn't the certificate supposed to renew automatically?\"",[11,20,21],{},"Incidents like this are far from rare in K3s and Kubernetes environments. Installing cert-manager and having cert-manager actually keep renewing certificates are two completely different problems. The old manual process — generating a CSR, submitting it to a certificate authority, receiving the signed cert, and uploading it — used to break down every time the responsible engineer changed and the handoff got lost. Ironically, even after automation tools are introduced, the same tribal-knowledge risk lingers in a new form: misconfiguration and missing monitoring.",[11,23,24,25,32],{},"K3s ",[26,27,31],"a",{"href":28,"rel":29},"https:\u002F\u002Fdocs.k3s.io\u002Fnetworking\u002Fnetworking-services",[30],"nofollow","bundles Traefik as its default Ingress controller",", and TLS termination at the edge relies on the combination of Traefik and the Secret that cert-manager issues. That means if cert-manager's renewal stalls, TLS termination for the entire K3s cluster stops with it.",[11,34,35],{},"This post walks through four patterns where certificates silently expire even with cert-manager installed, and the design needed to turn TLS certificate auto-renewal from something you \"installed\" into something that's actually \"running,\" in a K3s context.",[37,38,40],"h2",{"id":39},"why-installed-and-working-are-different-problems-four-ways-cert-manager-fails-silently","Why \"Installed\" and \"Working\" Are Different Problems — Four Ways cert-manager Fails Silently",[11,42,43],{},[44,45],"img",{"alt":46,"src":47},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-cert-manager-acme-renewal-pitfalls\u002Fsection01.webp",[11,49,50,51,56,57,62],{},"cert-manager is an ",[26,52,55],{"href":53,"rel":54},"https:\u002F\u002Fgithub.com\u002Fcert-manager\u002Fcert-manager",[30],"open-source project hosted on GitHub"," and a ",[26,58,61],{"href":59,"rel":60},"https:\u002F\u002Fwww.cncf.io\u002Fannouncements\u002F2024\u002F11\u002F12\u002Fcloud-native-computing-foundation-announces-cert-manager-graduation\u002F",[30],"CNCF Graduated project",", used widely enough to see 500 million downloads a month — a genuinely trusted tool. Yet certificate renewal failures in production remain common. The typical causes fall into four categories.",[64,65,67],"h3",{"id":66},"_1-issuer-clusterissuer-misreferences","1. Issuer \u002F ClusterIssuer misreferences",[11,69,70,71,74,75,78],{},"cert-manager defines a certificate's issuing authority as either an ",[15,72,73],{},"Issuer"," (namespace-scoped) or a ",[15,76,77],{},"ClusterIssuer"," (cluster-wide). If a Certificate resource references the wrong one, or a cross-namespace reference is misconfigured, issuance simply never happens. This kind of misconfiguration is especially common in production clusters spanning multiple namespaces.",[64,80,82],{"id":81},"_2-failed-dns-01-validation","2. Failed DNS-01 validation",[11,84,85,86,91],{},"For wildcard certificates or multi-domain certificates, cert-manager verifies domain ownership using the ",[26,87,90],{"href":88,"rel":89},"https:\u002F\u002Fcert-manager.io\u002Fdocs\u002Fconfiguration\u002Facme\u002Fdns01\u002F",[30],"DNS-01 challenge"," — writing a TXT record the ACME server requests, waiting for propagation, and then validating it. If you change an Ingress hostname and forget to update the corresponding DNS record, or if a DNS provider's API credentials expire, validation quietly keeps failing.",[64,93,95],{"id":94},"_3-lets-encrypt-rate-limit-exhaustion","3. Let's Encrypt rate limit exhaustion",[11,97,98,99,104],{},"Let's Encrypt enforces rate limits to keep usage fair. According to its ",[26,100,103],{"href":101,"rel":102},"https:\u002F\u002Fletsencrypt.org\u002Fdocs\u002Frate-limits\u002F",[30],"official documentation",", a registered domain can have up to 50 certificates issued per 7 days, and a single account can create up to 300 new orders every 3 hours. If a misconfiguration causes the same certificate request to fail repeatedly, these limits get hit fast, blocking even legitimate renewal requests. Worse, more than 5 authorization failures per hour trigger a temporary account-level restriction that requires action through the self-service portal to recover from.",[64,106,108],{"id":107},"_4-secrets-update-but-pods-and-ingress-dont-reload","4. Secrets update, but Pods and Ingress don't reload",[11,110,111,112,115,116,121,122,125],{},"When cert-manager renews a certificate, it writes the new content into a Kubernetes ",[15,113,114],{},"Secret",". Per the ",[26,117,120],{"href":118,"rel":119},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fconfiguration\u002Fsecret\u002F",[30],"Kubernetes documentation",", if a Secret is mounted as a volume, kubelet detects the change and eventually updates the file — but this doesn't apply when the Secret is mounted via ",[15,123,124],{},"subPath",". On top of that, applications that read certificates from environment variables, or processes that load a certificate into memory only once at startup, will keep using the stale certificate until they're restarted, even after the Secret updates.",[11,127,128],{},"What all four patterns share is that none of them surface as a visible error. Even when renewal fails, the system looks like it's working normally right up until the certificate's actual expiration. It's the same \"you don't notice until it's too late\" structure that manual certificate management always had.",[11,130,131,132,137],{},"Many of these pitfalls stem from rebuilding Issuer configuration and DNS provider integration from scratch every time a new cluster is created. With a managed K3s platform like ",[26,133,136],{"href":134,"rel":135},"https:\u002F\u002Fkubo.hexabase.io\u002F",[30],"Kubo",", where cert-manager installation and ClusterIssuer setup are templated in from the start, you begin from a state where reference errors and configuration gaps are structurally less likely to happen.",[37,139,141],{"id":140},"the-design-that-turns-should-be-working-into-is-working-monitoring-renewbefore-and-staging-validation","The Design That Turns \"Should Be Working\" Into \"Is Working\" — Monitoring, renewBefore, and Staging Validation",[11,143,144],{},[44,145],{"alt":146,"src":147},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-cert-manager-acme-renewal-pitfalls\u002Fsection02.webp",[11,149,150],{},"Keeping certificate auto-renewal actually functioning requires more than installing a tool — it takes a mechanism for detecting failure and a design with enough margin for error.",[64,152,154],{"id":153},"build-in-monitoring-that-catches-problems-before-expiration","Build in monitoring that catches problems before expiration",[11,156,157,158,161],{},"cert-manager exposes renewal status through the Certificate resource, which can be scraped as Prometheus metrics. It's important to dashboard remaining certificate validity and fire alerts once it drops below a threshold — independently of monitoring cert-manager itself. The key is monitoring \"whether the certificate is currently valid\" directly, rather than \"whether the last renewal succeeded.\" With a managed K3s platform like ",[26,159,136],{"href":134,"rel":160},[30],", which ships with Prometheus and Grafana out of the box, you don't need to build this monitoring stack from scratch.",[64,163,165],{"id":164},"secure-a-buffer-with-renewbefore","Secure a buffer with renewBefore",[11,167,168,169,174,175,178],{},"According to ",[26,170,173],{"href":171,"rel":172},"https:\u002F\u002Fcert-manager.io\u002Fdocs\u002Fusage\u002Fcertificate\u002F",[30],"cert-manager's official documentation",", when no renewal timing is specified, a certificate renews automatically once two-thirds of its validity period has elapsed — roughly 30 days before expiry for a 90-day certificate. This default acts as a buffer that absorbs transient failures from DNS propagation delays or rate limits. If you set ",[15,176,177],{},"spec.renewBefore"," explicitly, be careful not to shrink that buffer too much.",[11,180,181,182,187],{},"Note that Let's Encrypt has ",[26,183,186],{"href":184,"rel":185},"https:\u002F\u002Fletsencrypt.org\u002F2026\u002F02\u002F24\u002Frate-limits-45-day-certs",[30],"publicly announced"," a phased plan to shorten default certificate lifetimes, moving from 90 days to 64 days and eventually to 45 days. As lifetimes shrink, the window between a failed renewal and real production impact shrinks too — making monitoring and buffer design even more important.",[64,189,191],{"id":190},"validate-in-staging-before-applying-to-production","Validate in staging before applying to production",[11,193,194],{},"Issuer configuration changes or new DNS provider integrations should be validated against Let's Encrypt's staging environment before switching over the ClusterIssuer. Staging enforces the same rate-limiting mechanics as production, but failures there don't consume your production rate limit. Iterating on a misconfiguration directly against the production ClusterIssuer can burn through your rate limit all by itself.",[37,196,198],{"id":197},"a-managed-k3s-platform-can-eliminate-expired-without-anyone-noticing-structurally","A Managed K3s Platform Can Eliminate \"Expired Without Anyone Noticing\" Structurally",[11,200,201],{},[44,202],{"alt":203,"src":204},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-cert-manager-acme-renewal-pitfalls\u002Fsection03.webp",[11,206,207],{},"None of the four failure patterns covered here, or the monitoring and buffer design that prevents them, are technically difficult on their own. But keeping all of it \"correctly configured and correctly monitored\" continuously is itself an easily overlooked operational cost. Reviewing Issuer configuration, maintaining Prometheus alert rules, rotating DNS credentials — this kind of unglamorous, ongoing work becomes more prone to tribal knowledge as the number of K3s clusters grows, or as the people responsible for them change.",[11,209,210,213],{},[26,211,136],{"href":134,"rel":212},[30],"'s managed K3s platform ships with certificate automation via cert-manager and Let's Encrypt built in. There's no need to build and maintain ClusterIssuer setup, DNS provider integration, or monitoring from scratch, and you still get the full feature set of standard Kubernetes on top of K3s's lightweight footprint. Running a container platform equivalent to EKS or AKS at roughly ¥48,000\u002Fmonth for a 4vCPU\u002F8GB\u002F40GB × 3-node configuration is possible in large part because this kind of operational work is standardized.",[11,215,216],{},"Certificate management is a deceptively unglamorous area with a specific property: the damage when it breaks is large, but it's hard to notice until it does. Handing this off structurally to a managed platform isn't just about convenience — it's a design decision that reduces the actual risk of production incidents.",[37,218,220],{"id":219},"summary","Summary",[11,222,223],{},"Between installing cert-manager and having TLS certificate auto-renewal actually keep working lies a set of hard-to-see pitfalls: Issuer misreferences, failed DNS-01 validation, Let's Encrypt rate limit exhaustion, and Secrets that never get reloaded. Automation was supposed to eliminate the tribal-knowledge risk of manual renewal, but neglecting configuration and monitoring drags you right back into the same \"you don't notice until it's too late\" structure.",[11,225,226,227,230],{},"Monitoring expiration directly, securing a sufficient buffer with renewBefore, and validating changes in staging — only once these three practices are built into your operations does certificate management actually move from \"installed\" to \"working.\" And if you want to eliminate this operational burden structurally, migrating to a managed K3s platform like ",[26,228,136],{"href":134,"rel":229},[30],", where cert-manager operations come standard, is a realistic option worth considering.",{"title":232,"searchDepth":233,"depth":233,"links":234},"",2,[235,242,247,248],{"id":39,"depth":233,"text":40,"children":236},[237,239,240,241],{"id":66,"depth":238,"text":67},3,{"id":81,"depth":238,"text":82},{"id":94,"depth":238,"text":95},{"id":107,"depth":238,"text":108},{"id":140,"depth":233,"text":141,"children":243},[244,245,246],{"id":153,"depth":238,"text":154},{"id":164,"depth":238,"text":165},{"id":190,"depth":238,"text":191},{"id":197,"depth":233,"text":198},{"id":219,"depth":233,"text":220},"2026-08-27","Even with cert-manager installed, K3s production certificates can silently expire due to Issuer misconfiguration, failed DNS validation, Let's Encrypt rate limits, or Secrets that never get reloaded. This post covers the monitoring and design choices that turn ACME auto-renewal from 'installed' into 'actually working.'","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-cert-manager-acme-renewal-pitfalls\u002Feyecatch.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-cert-manager-acme-renewal-pitfalls",{"title":5,"description":250},"blog\u002Fen\u002Fk3s-cert-manager-acme-renewal-pitfalls",[260,261,17,262,263,264],"k3s","kubernetes","tls","lets-encrypt","acme","wqZydk3e3AQcKiu0KWx9qaiFvWlBwaL9y7eEHbKpaNk",[267,276,282,293,302,312],{"path":268,"title":269,"description":270,"date":271,"tags":272},"\u002Fblog\u002Fen\u002Fcert-manager-automatic-tls","Automating TLS Certificate Management in Kubernetes with cert-manager","Automate TLS certificate issuance and renewal in Kubernetes using cert-manager. Covers Let's Encrypt, ACME challenges, Ingress integration, and best practices.","2026-05-27",[17,261,262,263,264,273,274,275],"cncf","security","certificates",{"path":277,"title":278,"description":279,"date":280,"tags":281},"\u002Fblog\u002Fen\u002Fkubernetes-certificate-management-cert-manager-process-debt","The Cert Renewal Took One Line of Code and Two Months of Meetings: Why Kubernetes Certificate Management Is a Process Problem, Not a Technical One","Kubernetes certificate management is technically a matter of days. What actually takes time is the organizational process of getting sign-off. Here's how cert-manager automates the technical side, and how to design away the operational debt that remains.","2026-08-09",[260,261,17,262,274],{"path":283,"title":284,"description":285,"date":286,"tags":287},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[260,261,288,289,290,291,292],"storage","local-path-provisioner","persistent-volume","node-affinity","zfs",{"path":294,"title":295,"description":296,"date":297,"tags":298},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[260,261,299,300,301],"hpa","keda","autoscaling",{"path":303,"title":304,"description":305,"date":306,"tags":307},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[260,261,308,309,310,311],"proxmox","etcd","live-migration","high-availability",{"path":313,"title":314,"description":315,"date":316,"tags":317},"\u002Fblog\u002Fen\u002Fk3s-harbor-private-registry-docker-hub-rate-limit","Our CI Died at Midnight from Docker Hub's Pull Rate Limit. Designing a Harbor Private Registry for K3s\u002FKubo","Ever had a K3s cluster blocked by Docker Hub's pull rate limit? Here's how a Harbor private registry with Proxy Cache dodges the limit, plus vulnerability scanning, signing, and production-grade operational design.","2026-09-08",[260,261,318,319,320],"harbor","container-registry","private-registry",1792488901826]