Compunnel Digital

LLM & AI Agent Observability: Tools and Best Practices for 2027

LLM and AI agent observability goes beyond uptime and latency, because a model can respond fast and still be wrong. This guide covers what to instrument across the agent trace, which quality, cost and safety signals to alert on, where OpenTelemetry's GenAI standard stands in 2026, and how to choose the right platform for each job.

Blogs

  • Applied AI
  • Cloud Platforms & Engineering

Most enterprises now have AI features in production and very little idea whether they are working.

13%
of reliability professionals described themselves as very or extremely confident in their ability to monitor AI reliability.
Source: Catchpoint SRE Report 2026

The applications shipped. The instrumentation did not.

This guide is about closing that gap. It covers what makes LLM and agent observability different from application monitoring, what to instrument, where the OpenTelemetry standard actually stands, which platforms serve which job, and the practices that separate teams who can debug an agent from teams who can only restart it.

Key takeaways

  • LLM observability is a quality problem as much as a performance problem. A model call can return in 200 milliseconds, cost nothing unusual, and still be wrong, which means latency and error-rate monitoring alone will report a healthy system.
  • Gartner’s Market Guide for AI Evaluation and Observability Platforms, published 2 February 2026, forecasts that 60% of software engineering teams will use these platforms by 2028, up from 18% in 2025.
  • Agents raise the difficulty again. A single user request becomes a tree of model calls, tool calls and retries, so the unit of investigation is the whole run rather than one request.
  • OpenTelemetry now has GenAI semantic conventions covering model spans, agent spans, metrics and events, with provider-specific conventions for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference. They sit at Development status, so expect the attribute names to keep moving.
  • Evaluation is the part teams skip and then need most. Running scored evaluations against versioned datasets in the deployment pipeline is what makes a prompt or model change safe to ship.

Why LLM observability is a different problem

Traditional observability answers whether a system is up and how fast it responded. Both questions have objective answers. A request either returned a 200 or it did not. For a language model, the equivalent questions have no single right answer, because the output is generated rather than retrieved, and two identical inputs can produce different responses.

That breaks the usual signal set in three ways. Correctness becomes a judgment rather than a status code, so something has to score the output.

Cost becomes variable per request, because tokens consumed depend on the input, the context attached and how much the model chose to say. And failure becomes partial: a response can be fluent, fast, cheap and wrong, which no threshold on latency or error rate will catch.

Gartner’s framing in its February 2026 Market Guide is that AI applications and agents are “inherently nondeterministic and opaque,” which is why the firm treats evaluation and observability as one market rather than two.

The guide defines AI evaluation and observability platforms as tools that automate the scoring of AI outputs against quality criteria such as accuracy, fairness and performance, and then feed logs, metrics and traces back into those evaluations.

Gartner forecasts 60% platform adoption by 2028 while only 41% of teams run OpenTelemetry in production and 13% feel confident monitoring AI reliability.
Gartner forecasts 60% platform adoption by 2028 while only 41% of teams run OpenTelemetry in production and 13% feel confident monitoring AI reliability.

The numbers show a field that knows what it needs and has not built it yet. Gartner’s adoption forecast moves from 18% in 2025 to 60% by 2028.

Grafana Labs’ 2026 Observability Survey, fielded between 1 October 2025 and 6 January 2026 across 1,363 practitioners in 76 countries, found 47% had increased their OpenTelemetry investment over the past year while only 41% actually run it in production. And the Catchpoint confidence figure sits at 13%. Instrumentation intent is running well ahead of instrumentation reality.

What to instrument: the agent trace tree

For a single model call, the useful record is one span carrying the model name, the operation, token counts for input and output, latency, finish reason, and the prompt and response content where your privacy rules allow it. That alone answers most debugging questions about a chatbot or a summarizer.

Agents need a tree. One user request triggers a planning step, several tool calls, possibly a sub-agent, retries where a tool failed, and a final synthesis.

Each of those is a span, and the parent-child relationships are the information you need most, because the question in an agent post-mortem is almost never “was the model slow.” It is “which step went wrong, and what did the steps before it hand to it.”

The agent trace tree: what to record at the session, run and step level
The agent trace tree: what to record at the session, run and step level

Three levels deserve their own treatment:

  • Session level: record the user request, the final outcome and whether a human intervened.
  • Run level: record the agent name, the number of steps taken, total tokens and total cost, and whether the run hit a step limit or a loop guard.
  • Step level: record each model call and each tool call with its arguments, its result and its duration.

Tool calls are where agents fail most visibly and where instrumentation is most often missing. An agent that calls a search API with a malformed query, gets an empty result, and then confidently reports that no records exist has produced a wrong answer with no model error anywhere in the trace. Only the tool span shows it.

The signals worth alerting on

Four families of signal cover most production needs.

Quality signals

Quality signals come from evaluations rather than from the application. These include scores from an LLM-as-a-judge evaluator, rule-based checks such as schema validation or refusal detection, and human review samples. They are the only signals that detect a response that is wrong rather than broken.

Cost signals

Cost signals are token counts per request, per user, per feature and per model, plus the run-level total for agents. Cost per successful outcome is more useful than cost per call, because an agent that retries five times to get one right answer looks cheap per call and expensive per result. Grafana’s survey found cost has been the top tool selection criterion for three consecutive years, named by 65% of respondents.

Performance signals

Performance signals are the familiar ones with a twist: time to first token matters more than total latency for anything streaming to a user, and step count matters for agents because a run that takes twelve steps instead of four is usually a reasoning failure rather than a slow one.

Safety signals

Safety signals cover prompt injection attempts, PII appearing in prompts or responses, and guardrail triggers. Gartner lists runtime guardrails that detect hallucinations, PII leakage and prompt injection among the differentiating capabilities of these platforms rather than the mandatory ones, which is a fair reflection of how unevenly they are implemented.

The 2026 platform landscape

Gartner’s February 2026 Market Guide names twenty representative vendors, which is a useful map of a market that did not exist in this shape two years ago. The platforms divide roughly by what they were built to do first.

PlatformBuilt aroundTypical fit
LangfuseOpen-source tracing with evaluation and prompt management on topTeams that want self-hosting and OTLP ingest
LangSmith (LangChain)Tracing and evaluation tied closely to LangChain and LangGraphTeams already building on LangChain
BraintrustEvaluation-first workflow with datasets, scorers and CI gatesTeams treating evals as part of the pipeline
Arize AIML observability heritage extended to LLM tracing and driftOrganizations with existing ML monitoring practice
Comet (Opik)Experiment tracking extended into LLM evaluation and tracingResearch-adjacent teams with model development in house
GalileoEvaluation and guardrails with agent-specific failure analysisAgent-heavy applications needing runtime checks
Fiddler AIModel monitoring and explainability with governance reportingRegulated industries needing audit evidence
Datadog LLM ObservabilityLLM traces inside an existing APM and infrastructure platformTeams whose operations already live in Datadog

The guide also names Airia, Confident AI, CoreWeave, Deepchecks, Evidently AI, HoneyHive, IBM, Maxim, Microsoft, Openlayer, Orq.ai, Patronus AI and Plurai. On the observability-platform side, Dynatrace, New Relic, Splunk and Weights and Biases Weave all ship LLM tracing into stacks that already carry the rest of the estate.

Choosing between them follows the same logic as the AIOps market. An organization whose operations team already works inside one observability platform usually gets more from that platform’s LLM module than from a better standalone tool nobody opens.

An organization whose AI work sits with a product or data science team usually gets more from an evaluation-first platform, because that team’s bottleneck is shipping prompt changes safely rather than watching dashboards.

Gartner sets a floor worth holding vendors to. Its mandatory capabilities include:

  • A/B and multivariate testing of prompts and models
  • Regression prevention when a model or prompt changes
  • Safety and alignment checks
  • Root-cause diagnosis from logs and traces
  • Automated evaluation runs against datasets using code-based, human or LLM-as-a-judge evaluators
  • Dataset management with versioning and ground-truth labelling

A platform missing dataset versioning cannot support a regression gate, whatever else it does well.

Where the OpenTelemetry standard actually stands

OpenTelemetry’s GenAI semantic conventions now live in their own repository, separate from the core semantic conventions, and they cover model spans, agent spans, metrics, events and exceptions, plus provider-specific conventions for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference, and a related set for Model Context Protocol.

The important detail for planning is the maturity label: Development. The conventions are real, widely implemented and worth adopting, and the attribute names are still subject to change.

Teams instrumenting now should route telemetry through the OpenTelemetry Collector rather than wiring SDKs directly to a vendor endpoint, so that a convention change or a vendor change becomes a collector configuration edit instead of a redeployment of every service.

This is the same portability argument that applies to the rest of the estate, and the same evidence supports it. Grafana’s respondents named ease of adoption (41%) and freedom to switch vendors (37%) as their top reasons for choosing OpenTelemetry.

Best practices for 2026

A workable sequence for a team with AI in production and no observability on it:

  1. Instrument traces before dashboards. Get model spans and tool spans emitting through OpenTelemetry with token counts and costs attached. Everything else is built on this.
  2. Decide the content policy early. Prompts and responses are the most useful debugging data and often the most sensitive. Settle what gets recorded, what gets redacted and what gets sampled before the traces accumulate, not after a review asks.
  3. Build one evaluation dataset from production traffic. Fifty real examples with known-good answers beats a thousand synthetic ones. Version it.
  4. Put the evaluation in the pipeline. A prompt or model change that drops the score on the dataset should fail the build. Gartner lists regression prevention as mandatory for a reason: without a gate, every prompt edit is a production experiment.
  5. Alert on cost per outcome and on run length, not just on error rate. These are the two signals that catch an agent degrading rather than failing.
  6. Keep a human in the loop on anything consequential, and show the reasoning. Grafana found 95% of practitioners consider it important that AI shows its reasoning, and 15% do not yet trust AI to take action at all.
Don’t skip step four. The step teams most often skip is the fourth, and it is the one that converts observability from a debugging aid into a release control.

How Compunnel Digital helps

Compunnel Digital builds the platform layer that AI observability depends on, and governs the models running on top of it.

  • Applied AI and Agentic Systems covers the AI Platform and LLMOps foundation for deploying, managing and optimizing LLM systems, along with Autonomous Operations and Agentic AI for multi-step agent workflows. AI-OS™ is the delivery platform for moving AI work from pilot to production, and AgentWeave™ and SearchCore™ provide the agent and retrieval foundations.
  • Responsible AI, Governance and Assurance addresses the security, compliance and trustworthiness side, with ModelGuard™ as the governance layer that ships with every Compunnel AI engagement.
  • Managed SRE and AIOps supplies the observability architecture underneath: metrics, logs, traces and operational visibility across environments, with ML-assisted event correlation, anomaly detection and root-cause analysis.
  • Cloud-native accelerators: PlatformX™ supplies the observability and platform engineering patterns, and ShieldCI™ embeds security gates and detection into CI/CD pipelines so evaluation and safety checks run as part of delivery.

AI observability and governance assessment

Have AI features in production and no agreed way to tell whether they are behaving?

Talk to our team

Summary

LLM and agent observability is a quality problem as much as a performance problem, because a model call can be fast, cheap and still wrong. The practical path has four parts: trace the whole agent run as a tree of model calls, tool calls and retries; alert on quality, cost, performance and safety signals; pick a platform by the job it needs to do, using Gartner’s February 2026 Market Guide as a map; and build on OpenTelemetry GenAI conventions for portability while accepting that they are still at Development status. Start with evaluation, the step teams skip most often, and expand from there.

Frequently asked questions

What is LLM observability?

LLM observability is the practice of collecting and interpreting the data needed to understand how a language model application behaves in production. It covers traces of model and tool calls, token and cost accounting, latency, and quality scores from evaluations. It differs from application monitoring because output correctness is a judgment rather than a status code, so scoring is part of the instrumentation.

How is AI agent observability different from LLM observability?

A single model call produces one span. An agent run produces a tree of spans: planning steps, tool calls, retries, sometimes sub-agents. Agent observability adds the run as a unit of analysis, with step counts, loop detection, per-run cost and the parent-child structure that shows which step failed and what it was given.

Does OpenTelemetry support LLM and agent tracing?

Yes. OpenTelemetry has GenAI semantic conventions covering model spans, agent spans, metrics and events, with provider-specific conventions for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference, maintained in a dedicated repository. They are at Development status, so attribute names can still change, and routing through the Collector limits the cost of that churn.

What are the best LLM observability tools?

It depends on which job comes first. Langfuse, LangSmith, Braintrust, Arize, Comet Opik, Galileo and Fiddler lead on tracing and evaluation, while Datadog, Dynatrace, New Relic and Splunk fold LLM traces into platforms that already hold the rest of the estate. Gartner’s February 2026 Market Guide names twenty representative vendors in the category.

What should we monitor first in an LLM application?

Traces with token counts and cost, then one evaluation dataset built from real traffic. Traces make incidents debuggable and evaluations make changes safe. Dashboards and alert thresholds are easier to design once both exist.

Conclusion

AI observability is where application observability was a decade ago: the standard exists, the tooling is consolidating, and most teams have shipped faster than they have instrumented. The gap is visible in the numbers, with 13% of reliability professionals confident they can monitor AI reliability against a Gartner forecast of 60% platform adoption by 2028. Closing it starts with traces through OpenTelemetry, a content policy decided in advance, one versioned evaluation dataset drawn from production, and an evaluation gate in the pipeline so a prompt change cannot quietly degrade a live feature.

Explore how our Applied AI and Agentic Systems practice builds governed AI platforms with observability designed in.

Sources

  • Kupparaju Harith Pranav

    Kupparaju Harith Pranav

    Technology Consultant | Digital Transformation | Healthcare & Life Sciences | Enterprise SolutioningCompunnel Digital

Stay in the loop

Get the latest insights, updates, and product news delivered to your inbox