[{"data":1,"prerenderedAt":287},["ShallowReactive",2],{"blog-en-k3s-ai-agent-opentelemetry-observability":3,"blog-related-en-k3s-ai-agent-opentelemetry-observability":232,"blog-en-k3s-ai-agent-opentelemetry-observability-alt":220},{"id":4,"title":5,"author":6,"body":7,"date":214,"description":215,"extension":216,"image":217,"locale":218,"meta":219,"navigation":220,"path":221,"seo":222,"stem":223,"tags":224,"__hash__":231},"blog\u002Fblog\u002Fen\u002Fk3s-ai-agent-opentelemetry-observability.md","Your AI Agent Is Running on K3s, But You Have No Idea What It's Doing. Tracing Non-Deterministic Execution with OpenTelemetry","Kubo Team",{"type":8,"value":9,"toc":205},"minimark",[10,15,23,31,38,54,58,64,73,82,85,89,95,104,113,131,139,143,149,162,171,186,190,193],[11,12,14],"h2",{"id":13},"_1-why-its-running-isnt-enough-anymore","1. Why \"It's Running\" Isn't Enough Anymore",[16,17,18],"p",{},[19,20],"img",{"alt":21,"src":22},"section01","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-agent-opentelemetry-observability\u002Fsection01.webp",[16,24,25,26,30],{},"Do you still think of ",[27,28,29],"strong",{},"AI agent observability"," as just bolting on one more monitoring tool? Teams running MCP servers and AI agents in production on K3s clusters quickly hit a wall: you can tell the service is running, but you have no idea what it was \"thinking\" when it decided to call a particular tool.",[16,32,33,34,37],{},"With a traditional microservice, a request flows from Service A to Service B in a fixed order. When something breaks, an existing APM tool can pinpoint the failure mechanically. AI agents are different. The LLM judges context on the fly and dynamically decides which tool to call, in what order, and how many times. This is called ",[27,35,36],{},"non-deterministic execution",", and you can't draw a flowchart in advance to define your monitoring checkpoints.",[16,39,40,41,45,46,53],{},"Adding more structured logs doesn't fix this — you can confirm \"a tool was called,\" but you still can't trace the chain of reasoning behind ",[42,43,44],"em",{},"why"," that tool was chosen. This is the gap that OpenTelemetry's GenAI (generative AI) semantic conventions are being standardized to fill. If you're already running on a K3s-based platform, the monitoring foundation is often built in from day one, leaving you to focus purely on adding the tracing layer (",[47,48,52],"a",{"href":49,"rel":50},"https:\u002F\u002Fkubo.hexabase.io\u002F",[51],"nofollow","Kubo",", for instance, ships with Prometheus + Grafana out of the box).",[11,55,57],{"id":56},"_2-what-are-the-opentelemetry-genai-semantic-conventions","2. What Are the OpenTelemetry GenAI Semantic Conventions?",[16,59,60],{},[19,61],{"alt":62,"src":63},"section02","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-agent-opentelemetry-observability\u002Fsection02.webp",[16,65,66,67,72],{},"According to the ",[47,68,71],{"href":69,"rel":70},"https:\u002F\u002Fopentelemetry.io\u002Fblog\u002F2025\u002Fai-agent-observability\u002F",[51],"official OpenTelemetry blog",", the rapid adoption of AI agents makes a unified telemetry format essential for interoperability across frameworks. If every vendor uses its own logging format, you end up rebuilding your entire monitoring stack every time you switch agent frameworks.",[16,74,75,76,81],{},"Standardization is progressing in two stages. The application layer for agents has already been finalized, building on proposals Google published around AI agents. Meanwhile, the common conventions covering major frameworks like CrewAI, AutoGen, and LangGraph are still under active development. The ",[47,77,80],{"href":78,"rel":79},"https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002F",[51],"OpenTelemetry GenAI semantic conventions documentation"," is progressively formalizing agent spans, per-provider LLM conventions, event\u002Fexception\u002Fmetric definitions, and even how to describe tool calls like those in MCP.",[16,83,84],{},"The key point is that telemetry here isn't just a failure-detection mechanism — it's designed as a feedback loop for continuously improving agent response quality. Recording not just \"is it running\" but \"how did it reason\" is becoming part of what it means to operate an AI agent in production.",[11,86,88],{"id":87},"_3-tracing-mcp-server-tool-calls","3. Tracing MCP Server Tool Calls",[16,90,91],{},[19,92],{"alt":93,"src":94},"section03","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-agent-opentelemetry-observability\u002Fsection03.webp",[16,96,97,98,103],{},"The ",[47,99,102],{"href":100,"rel":101},"https:\u002F\u002Fmodelcontextprotocol.io\u002Fintroduction",[51],"official Model Context Protocol (MCP) documentation"," compares the standard that connects AI applications to external systems to a USB-C port. MCP has three components — host, client, and server — and an AI agent calls tools like databases or external APIs through this standardized interface.",[16,105,106,107,112],{},"The problem is that MCP servers themselves have no observability built in. ",[47,108,111],{"href":109,"rel":110},"https:\u002F\u002Fsignoz.io\u002Fblog\u002Fmcp-observability-with-otel\u002F",[51],"SigNoz's write-up on MCP observability"," frames this as a \"black box\" problem: when an external API's latency degrades the agent's overall response time, or a tool call silently returns invalid data, you can't tell which component in the distributed system is actually responsible.",[16,114,115,116,121,122,126,127,130],{},"A practical answer to this is context propagation based on the ",[47,117,120],{"href":118,"rel":119},"https:\u002F\u002Fwww.w3.org\u002FTR\u002Ftrace-context\u002F",[51],"W3C Trace Context specification",". The ",[123,124,125],"code",{},"traceparent"," header carries the trace ID and parent span ID in a fixed-length format, while ",[123,128,129],{},"tracestate"," chains vendor-specific information alongside it. Together, they let you stitch together a single trace spanning the MCP host, client, server, and any downstream external APIs.",[16,132,133,134,138],{},"When implementing this on K3s, it's usually easiest to deploy the OTel Collector as a cluster-wide DaemonSet rather than a sidecar, collecting traces from each MCP server pod. On a K3s-based platform like Kubo, which supports Helm charts and GitOps, you can add the OTel Collector as a Helm release into your ",[47,135,137],{"href":49,"rel":136},[51],"existing deployment flow"," without disrupting your operations.",[11,140,142],{"id":141},"_4-correlating-traces-with-kubernetes-metrics","4. Correlating Traces with Kubernetes Metrics",[16,144,145],{},[19,146],{"alt":147,"src":148},"section04","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-agent-opentelemetry-observability\u002Fsection04.webp",[16,150,151,152,157,158,161],{},"Traces alone can't tell you whether \"the LLM response is slow\" because inference itself is slow, or because the pod is under resource pressure. This is where Kubernetes' standard resource metrics pipeline comes in. As the ",[47,153,156],{"href":154,"rel":155},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fdebug\u002Fdebug-cluster\u002Fresource-metrics-pipeline\u002F",[51],"official Kubernetes documentation"," describes, cAdvisor collects container CPU and memory usage, kubelet passes it along, metrics-server aggregates it, and you can check it via the Metrics API through commands like ",[123,159,160],{},"kubectl top pods",".",[16,163,164,165,170],{},"Cross-referencing pod metrics against the agent execution traces collected via OTel, on the same timeline, reveals correlations like \"pod memory usage spiked right before this LLM call.\" In a real-world case ",[47,166,169],{"href":167,"rel":168},"https:\u002F\u002Fwww.cncf.io\u002Fblog\u002F2026\u002F06\u002F17\u002Fwhy-cloud-native-belongs-at-the-heart-of-agentic-ai-lessons-from-building-a-multi-agent-security-platform-on-kubernetes\u002F",[51],"published by the CNCF",", a team used the trace_id propagated via the A2A protocol to unify each agent's structured logs, Prometheus metrics, and network flow visualization — and traced an entire autonomous decision's reasoning chain in roughly 15 minutes.",[16,172,173,174,179,180,185],{},"Metrics collection commonly relies on a pull-model time-series database like ",[47,175,178],{"href":176,"rel":177},"https:\u002F\u002Fprometheus.io\u002Fdocs\u002Fintroduction\u002Foverview\u002F",[51],"Prometheus",". Because PromQL can flexibly aggregate multi-dimensional labeled data, embedding the trace ID as a label lets you extract only the metrics tied to a specific agent execution. Kubo Cloud ships with Prometheus + Grafana standard, so you don't need to build this correlation foundation from scratch. And for an open-architecture AI agent platform built around MCP\u002FSkills, like ",[47,181,184],{"href":182,"rel":183},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fcaptain-ai\u002F",[51],"Captain.AI",", this trace-metrics correlation design carries over directly.",[11,187,189],{"id":188},"_5-conclusion","5. Conclusion",[16,191,192],{},"Running AI agents and MCP servers in production is shifting from simply confirming \"is the service up\" to visualizing \"what did the agent think, and how did it act.\" OpenTelemetry's GenAI semantic conventions are still evolving, but combining W3C Trace Context propagation with Kubernetes' standard metrics pipeline already resolves many of these \"black box\" problems.",[16,194,195,196,199,200,161],{},"If you're putting AI agents on a K3s-based platform, observability design isn't something you can defer. If you'd like to talk through combining a standard monitoring stack with OTel tracing, reach out via ",[47,197,52],{"href":49,"rel":198},[51],"'s ",[47,201,204],{"href":202,"rel":203},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[51],"contact page",{"title":206,"searchDepth":207,"depth":207,"links":208},"",2,[209,210,211,212,213],{"id":13,"depth":207,"text":14},{"id":56,"depth":207,"text":57},{"id":87,"depth":207,"text":88},{"id":141,"depth":207,"text":142},{"id":188,"depth":207,"text":189},"2026-09-06","MCP servers and AI agents on K3s don't reveal why they called a given tool just from logs. Here's how OpenTelemetry's GenAI semantic conventions make non-deterministic execution observable.","md","https:\u002F\u002Fcdn.kubo.hexabase.io\u002Fimages\u002Fblog\u002Fk3s-ai-agent-opentelemetry-observability\u002Feyecatch.webp","en",{},true,"\u002Fblog\u002Fen\u002Fk3s-ai-agent-opentelemetry-observability",{"title":5,"description":215},"blog\u002Fen\u002Fk3s-ai-agent-opentelemetry-observability",[225,226,227,228,229,230],"k3s","kubernetes","opentelemetry","distributed-tracing","genai","telemetry","ieqR2Y_MpdSjL8bsbESm-gCmMztdvOBMQyRqykecqyA",[233,240,247,257,268,277],{"path":234,"title":235,"description":236,"date":237,"tags":238},"\u002Fblog\u002Fen\u002Fk3s-opentelemetry-observability-correlation","Stop Bouncing Between Three Dashboards: Correlating K3s Observability with OpenTelemetry","Are you burning incident-response time checking Prometheus, Loki, and Jaeger separately? Learn how to correlate metrics, logs, and traces with OpenTelemetry to cut investigation time on K3s clusters, with practical Collector deployment patterns.","2026-08-20",[225,226,227,239,228],"observability",{"path":241,"title":242,"description":243,"date":244,"tags":245},"\u002Fblog\u002Fen\u002Fkubernetes-aiops-incident-response-distributed-tracing","Telling AI to 'Fix It' Won't Work: Kubernetes Incident Response Starts With Distributed Tracing","Handing off Kubernetes incident response to AI doesn't help if the causal chain across a distributed system stays invisible. This piece covers the distributed tracing foundation that makes AIOps actually work, plus practical steps for adopting OpenTelemetry.","2026-07-27",[226,225,239,228,227,246],"aiops",{"path":248,"title":249,"description":250,"date":251,"tags":252},"\u002Fblog\u002Fen\u002Fopentelemetry-distributed-tracing","Implementing Distributed Tracing with OpenTelemetry: From Basics to Production","A practical guide to distributed tracing with OpenTelemetry. Covers Collector setup, auto-instrumentation, Jaeger\u002FTempo integration, and Kubernetes best practices.","2026-05-27",[227,228,226,253,239,254,255,256],"cncf","jaeger","tempo","microservices",{"path":258,"title":259,"description":260,"date":261,"tags":262},"\u002Fblog\u002Fen\u002Fk3s-local-path-provisioner-capacity-node-affinity-pvc-pending","Your K3s Pod Won't Leave Pending. The 'Ignored Capacity' and Node Affinity Trap in Local Storage","A K3s PVC stuck in Pending often traces back to one fact: local-path-provisioner never enforces capacity limits. This post digs into hypervisor thin provisioning, Volume Node Affinity Conflicts, and the backup misconceptions hiding in K3s local storage design.","2026-09-12",[225,226,263,264,265,266,267],"storage","local-path-provisioner","persistent-volume","node-affinity","zfs",{"path":269,"title":270,"description":271,"date":272,"tags":273},"\u002Fblog\u002Fen\u002Fk3s-keda-vs-hpa-event-driven-autoscaling-decision","KEDA vs HPA on K3s: A Decision Framework for Event-Driven Autoscaling","When CPU\u002Fmemory-based HPA on K3s can't keep up with event-driven workloads, should you extend HPA or adopt KEDA, the CNCF-graduated project? Here's a 3-point decision framework, plus what to watch for on K3s.","2026-09-10",[225,226,274,275,276],"hpa","keda","autoscaling",{"path":278,"title":279,"description":280,"date":281,"tags":282},"\u002Fblog\u002Fen\u002Fk3s-proxmox-live-migration-etcd-quorum-affinity-design","Proxmox Live Migration Quietly Breaks K3s etcd Quorum: When Virtualization Collides with Raft","A deep dive into how live-migrating a K3s control plane VM on Proxmox can break etcd quorum, explained through heartbeat\u002Felection-timeout mechanics and Proxmox affinity design, with concrete operational countermeasures.","2026-09-09",[225,226,283,284,285,286],"proxmox","etcd","live-migration","high-availability",1792488901244]