Your ServiceMonitor YAML Is Correct. So Why Isn't Prometheus Scraping It? A K3s Diagnostic Guide to the 'Connection Points'

You wrote a ServiceMonitor YAML, applied it, and opened the Prometheus UI to check Targets — only to find "0/0 up" and nothing there. You searched for kubernetes service monitor, copied the official Kubernetes manifest example verbatim, and still no metrics show up. If you're an infrastructure engineer running Prometheus Operator on K3s, you've almost certainly hit this wall before.

There's a common pattern behind this failure. A single ServiceMonitor YAML only works if it successfully passes through all four connection points: "Prometheus CR → ServiceMonitor → Service → Endpoints → Pod." If even one of these links is broken, the whole chain fails silently — no error message, nothing. This article walks through these four connection points in the order you should actually suspect them, and rethinks K3s monitoring operations from the ground up.

1. Why It Doesn't Work Even Though "the YAML Is Correct"

ServiceMonitor isn't a standard Kubernetes resource — it's a Custom Resource Definition (CRD) added independently by Prometheus Operator. That means a successful kubectl apply doesn't guarantee that Prometheus itself actually recognizes that ServiceMonitor.

Prometheus fundamentally collects metrics via a "pull" model, and targets are discovered through service discovery or static configuration (Prometheus official documentation). ServiceMonitor is merely a mechanism for declaratively configuring this target discovery through Kubernetes labels. In other words, whether "the labels are propagating correctly" holds the key to everything.

This structure resembles troubleshooting a multi-layered architecture where a request passes from frontend to backend to database. Each individual configuration might be correct, but unless you check each "connection point" across layers one by one, you can't pinpoint where processing is actually stalling. Troubleshooting ServiceMonitor works the same way — you need to suspect the following four connection points in order.

2. Connection Point 1: The Prometheus CR's serviceMonitorSelector

The first thing to suspect is the configuration of the Prometheus CRD itself. The Prometheus object defines which ServiceMonitors it targets via the spec.serviceMonitorSelector field (Prometheus Operator official documentation).

Unless this selector is empty (meaning "select all"), Prometheus Operator won't even notice a ServiceMonitor exists if it lacks the corresponding label. In fact, Prometheus Operator's own official troubleshooting guide explicitly states that you should first check "whether the ServiceMonitor is actually included in the generated Prometheus configuration (stored in a Secret named prometheus-<name>)" (prometheus-operator.dev troubleshooting guide).

Example diagnostic commands:

kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
kubectl get servicemonitor -A --show-labels

Also check serviceMonitorNamespaceSelector. If the ServiceMonitor is placed in a different namespace than the monitored workload, an overly narrow scope here will cause it to be ignored as out-of-scope even if the labels match.

3. Connection Point 2: ServiceMonitor's selector vs. the Service's Labels

Even after clearing connection point 1, you need to verify that the ServiceMonitor itself is correctly pointing at the right Service. ServiceMonitor's spec.selector is a mechanism for selecting target Service objects by label. The official documentation gives an example of "a ServiceMonitor that selects all Service objects with the label app: example-app" (as cited above, getting-started guide).

A classic mistake here happens when a ServiceMonitor is copy-pasted and only the matchLabels values are forgotten to be updated. Cross-check both sides like this:

kubectl get svc <service-name> -n <namespace> --show-labels
kubectl get servicemonitor <name> -n <namespace> -o yaml | grep -A5 selector

Once you've confirmed the label keys and values match exactly on both sides, move to the next layer.

4. Connection Point 3: The Easily-Overlooked Trap of Port Name Mismatches

This is the point people trip over most often. In Kubernetes, naming a Service's port (the name field) is optional by spec. But ServiceMonitor only references the port name — never the port number (as cited above, prometheus-operator.dev troubleshooting guide).

Here's how named ports work according to the official Kubernetes documentation (Kubernetes official: Service):

# Service side
ports:
  - name: metrics
    port: 9090
    targetPort: metrics
---
# ServiceMonitor side
endpoints:
  - port: metrics   # references the name, not the port number

This design exists so that client-side configuration won't break even if the port number a Pod exposes changes in the future (as cited above, Kubernetes official: Service). But the flip side is that if you forget to name the port on the Service side, or write a numeric port value (like 9090) directly on the ServiceMonitor side, the configuration looks correct but silently fails.

5. Connection Point 4: namespaceSelector and RBAC — the Invisible Permission Wall

If labels and port names all match but you still can't get metrics, what's left is a permission wall. If the Prometheus ServiceAccount hasn't been granted permission (get, list, watch) to fetch services and endpoints in the target namespace, discovery will silently fail even with a perfect label selector match. Prometheus Operator's own official troubleshooting guide lists this as a "common problem" (prometheus-operator/prometheus-operator troubleshooting guide).

According to the official Kubernetes RBAC documentation, the basic structure for granting these permissions via a ClusterRole looks like this (Kubernetes official: RBAC):

apiGroups: [""]
resources: ["services", "endpoints", "pods"]
verbs: ["get", "list", "watch"]

If you're using a Helm chart like kube-prometheus-stack, this configuration is usually auto-generated. But it's easy to overlook when you later add monitoring targets that span multiple namespaces, or when you've customized RBAC yourself. Checking kubectl get events for permission-denied events is also useful here.

There's also a K3s-specific gotcha to watch for. Unlike RKE2 or kubeadm-based clusters, K3s doesn't expose etcd or Traefik metrics by default (Rancher official documentation: How Monitoring Works). At this stage, it's worth ruling out the K3s-specific case where "the ServiceMonitor configuration is correct, but that component simply doesn't expose a metrics endpoint in the first place."

Summary — Pay the Diagnostic Cost Every Time, or Let It Come Standard?

Let's recap the diagnostic order:

1. Does the Prometheus CR's serviceMonitorSelector / serviceMonitorNamespaceSelector match the ServiceMonitor's labels and namespace?
2. Does the ServiceMonitor's spec.selector match the Service's labels?
3. Does spec.endpoints[].port reference the port name explicitly set on the Service side (not the port number)?
4. Does the Prometheus ServiceAccount have get/list/watch permissions on services/endpoints in the target namespace, and is this a case of K3s-specific unexposed metrics?

Prometheus sits at the core of the more than 38 CNCF Graduated projects (as of August 2026), a mature and battle-tested observability foundation (CNCF official: Prometheus). Precisely because the mechanism itself is proven and stable, the root cause of any failure always lies somewhere in the "connection points of declarative configuration." Before rushing to recreate everything from scratch, mechanically working through these four stages is the fastest path forward. Ninety percent of the time, the reason a ServiceMonitor isn't working lies in one of these four connection points. Check them one by one, methodically, and you can steadily eliminate mysterious failures.

That said, this diagnostic cost itself is a hidden burden of operating K3s. Repeating the same checklist every time you spin up a new cluster is hardly productive work. Kubo's K3s-based managed clusters come with a Prometheus + Grafana monitoring stack built in standard, so you never have to wire up ServiceMonitor from scratch.

At a price point that's 60% or less of what you'd pay monthly for EKS or AKS, Kubo Cloud offers a way to cut the time spent wrestling with these "broken connection points" altogether. Whether you keep running this diagnostic flow by hand every time, or hand the whole monitoring stack over to a managed K3s starting at ¥48,000/month — it's worth considering where your operations team's time is best spent.

If your organization's security requirements demand on-premises operation, Kubo On-Premise lets you build the same monitoring stack in an air-gapped environment. Preserving data sovereignty while handing off the complexity of these connection points to Kubo is also an option worth exploring — get in touch to discuss it.