1. Why "It's Running" Isn't Enough Anymore
Do you still think of AI agent observability as just bolting on one more monitoring tool? Teams running MCP servers and AI agents in production on K3s clusters quickly hit a wall: you can tell the service is running, but you have no idea what it was "thinking" when it decided to call a particular tool.
With a traditional microservice, a request flows from Service A to Service B in a fixed order. When something breaks, an existing APM tool can pinpoint the failure mechanically. AI agents are different. The LLM judges context on the fly and dynamically decides which tool to call, in what order, and how many times. This is called non-deterministic execution, and you can't draw a flowchart in advance to define your monitoring checkpoints.
Adding more structured logs doesn't fix this — you can confirm "a tool was called," but you still can't trace the chain of reasoning behind why that tool was chosen. This is the gap that OpenTelemetry's GenAI (generative AI) semantic conventions are being standardized to fill. If you're already running on a K3s-based platform, the monitoring foundation is often built in from day one, leaving you to focus purely on adding the tracing layer (Kubo, for instance, ships with Prometheus + Grafana out of the box).
2. What Are the OpenTelemetry GenAI Semantic Conventions?
According to the official OpenTelemetry blog, the rapid adoption of AI agents makes a unified telemetry format essential for interoperability across frameworks. If every vendor uses its own logging format, you end up rebuilding your entire monitoring stack every time you switch agent frameworks.
Standardization is progressing in two stages. The application layer for agents has already been finalized, building on proposals Google published around AI agents. Meanwhile, the common conventions covering major frameworks like CrewAI, AutoGen, and LangGraph are still under active development. The OpenTelemetry GenAI semantic conventions documentation is progressively formalizing agent spans, per-provider LLM conventions, event/exception/metric definitions, and even how to describe tool calls like those in MCP.
The key point is that telemetry here isn't just a failure-detection mechanism — it's designed as a feedback loop for continuously improving agent response quality. Recording not just "is it running" but "how did it reason" is becoming part of what it means to operate an AI agent in production.
3. Tracing MCP Server Tool Calls
The official Model Context Protocol (MCP) documentation compares the standard that connects AI applications to external systems to a USB-C port. MCP has three components — host, client, and server — and an AI agent calls tools like databases or external APIs through this standardized interface.
The problem is that MCP servers themselves have no observability built in. SigNoz's write-up on MCP observability frames this as a "black box" problem: when an external API's latency degrades the agent's overall response time, or a tool call silently returns invalid data, you can't tell which component in the distributed system is actually responsible.
A practical answer to this is context propagation based on the W3C Trace Context specification. The traceparent header carries the trace ID and parent span ID in a fixed-length format, while tracestate chains vendor-specific information alongside it. Together, they let you stitch together a single trace spanning the MCP host, client, server, and any downstream external APIs.
When implementing this on K3s, it's usually easiest to deploy the OTel Collector as a cluster-wide DaemonSet rather than a sidecar, collecting traces from each MCP server pod. On a K3s-based platform like Kubo, which supports Helm charts and GitOps, you can add the OTel Collector as a Helm release into your existing deployment flow without disrupting your operations.
4. Correlating Traces with Kubernetes Metrics
Traces alone can't tell you whether "the LLM response is slow" because inference itself is slow, or because the pod is under resource pressure. This is where Kubernetes' standard resource metrics pipeline comes in. As the official Kubernetes documentation describes, cAdvisor collects container CPU and memory usage, kubelet passes it along, metrics-server aggregates it, and you can check it via the Metrics API through commands like kubectl top pods.
Cross-referencing pod metrics against the agent execution traces collected via OTel, on the same timeline, reveals correlations like "pod memory usage spiked right before this LLM call." In a real-world case published by the CNCF, a team used the trace_id propagated via the A2A protocol to unify each agent's structured logs, Prometheus metrics, and network flow visualization — and traced an entire autonomous decision's reasoning chain in roughly 15 minutes.
Metrics collection commonly relies on a pull-model time-series database like Prometheus. Because PromQL can flexibly aggregate multi-dimensional labeled data, embedding the trace ID as a label lets you extract only the metrics tied to a specific agent execution. Kubo Cloud ships with Prometheus + Grafana standard, so you don't need to build this correlation foundation from scratch. And for an open-architecture AI agent platform built around MCP/Skills, like Captain.AI, this trace-metrics correlation design carries over directly.
5. Conclusion
Running AI agents and MCP servers in production is shifting from simply confirming "is the service up" to visualizing "what did the agent think, and how did it act." OpenTelemetry's GenAI semantic conventions are still evolving, but combining W3C Trace Context propagation with Kubernetes' standard metrics pipeline already resolves many of these "black box" problems.
If you're putting AI agents on a K3s-based platform, observability design isn't something you can defer. If you'd like to talk through combining a standard monitoring stack with OTel tracing, reach out via Kubo's contact page.