The Complete Agent Stack · Part 12 of 14 · Improvement loop layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 11: How to Evaluate an AI Data Agent · Next: Part 13: Golden Questions for AI Agent Evals →

Part 12 opens the improvement loop with production tracing and live LLM-as-judge evals. Its companion, Part 13 on golden questions, covers the other half of evaluation: fixed test cases and confidence scores.

Key Takeaways

• Agent telemetry now has an open vocabulary. OpenTelemetry's GenAI semantic conventions define spans for agent invocations, retrievals and tool calls, plus a gen_ai.evaluation.result event that attaches an eval score to the operation it judged. All of them are still marked as in development (OpenTelemetry).

• Strong LLM judges agree with people about as often as people agree with each other. In the MT-Bench and Chatbot Arena study, GPT-4 judges matched human preferences more than 80% of the time, the same level of agreement measured between human raters (Zheng et al., NeurIPS 2023).

• Judges carry measurable biases. Changing only the order in which two answers appeared let Vicuna-13B beat ChatGPT on 66 of 80 queries when ChatGPT was the evaluator, so a production judge needs checking against human labels (Wang et al.).

• A repeated question is a reliable frustration signal. Microsoft Research found that in search, a similar query issued within a short interval, such as five minutes, implies the user was dissatisfied with the first result (Hassan et al., CIKM 2013).

The 45-second version:

What Is AI Agent Observability?

AI agent observability is the combination of two things: a step-by-step record of how an agent produced each answer, and an ongoing assessment of whether those answers were any good. For a data agent, the steps include the context it retrieved, the skill or tool it chose, the SQL it wrote, whether that SQL passed validation, the rows it got back and the chart it rendered.

Traditional application monitoring answers whether a service is up and fast. An agent can return a response in two seconds, with no error, containing the wrong number, so agent observability adds a trace of reasoning and tool use on top of infrastructure metrics, and a quality judgement on top of the trace.

A session, or conversation, is a sequence of turns with one user. A trace follows one request from question to answer, and a span is one step inside it, such as a model call or a tool execution. OpenTelemetry ties spans to their conversation with a gen_ai.conversation.id attribute, which is what lets you read one user's retries in order (OpenTelemetry).

Why Do Data Agents Fail Quietly?

When a data agent gets something wrong, the usual result is a plausible answer that looks like every other answer. A query that uses the wrong revenue definition, filters on a stale status value or joins one table too many still returns a clean, specific figure. Our semantic layer guide explains why ungrounded agents produce confident wrong numbers instead of refusals.

Several common failure modes never raise an exception:

• Empty or null results. The query filters on a value that does not exist, returns nothing, and the agent reports that there is no data.

• Lazy answers. The agent does less than asked, such as answering in text when the user wanted a chart.

• Unanswerable questions. The user asks something the current data model cannot support, and the agent improvises.

• Silent definition choices. The agent picks one of several reasonable meanings of a business term without saying so.

Users who get a bad answer tend to ask again and then stop using the agent, often without reporting anything. Without traces, a builder also cannot tell whether the cause was the data, the context or the question, and each needs a different fix.

What Should an AI Agent Trace Capture?

A useful trace records one span per meaningful step, with enough detail to replay the agent's decisions. OpenTelemetry's GenAI conventions offer a vendor-neutral vocabulary, defining span types for model inference, embeddings, retrievals, memory operations and tool execution, and agent-level spans for creating and invoking agents, running workflows, planning and loading skills (OpenTelemetry). A tool call span, for example, should be named execute_tool followed by the tool's name.

Conversation and User

Record who asked, through which surface and in which conversation, so that one user's repeated attempts across an embedded app, Slack or an AI assistant appear as one history.

Retrieved Context

Record which definitions, documents and examples the agent retrieved before answering. When an answer is wrong, the first diagnostic question is whether the right definition ever reached the model.

Skills and Tool Calls

Record which skill the agent loaded, which tools it called, the arguments it passed and what came back. The conventions provide gen_ai.tool.call.arguments and gen_ai.tool.call.result attributes for this, both as opt-in fields.

SQL, Validation and Result Shape

For a data agent, this span carries most of the diagnostic value. Store the exact SQL, whether it passed validation, how many rows it returned and how many values were null. A zero-row result on a question that should have data is one of the cheapest and strongest failure signals available.

Output and Cost

Record the final text or chart, any rendering error, and token usage through attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Cost per trace shows which questions are expensive to answer and which conversations loop.

One caution applies to every span. OpenTelemetry classifies instructions, user messages and model outputs as sensitive, and recommends that instrumentations do not capture them by default and offer an opt-in instead (OpenTelemetry). For a data agent, both the questions and the result rows can contain customer data, so decide on retention and access rules before turning on full capture. Trace viewers should follow the same row-level security rules as the agent.

How Do LLM-as-Judge Evals Work on Live Usage?

A trace tells you what happened. An eval tells you whether it was good. Offline evals run a fixed test set before a change ships, and live evals score real conversations as they happen. Anthropic's guide to agent evals describes production monitoring as the method that reflects real user behaviour, with the drawback that problems reach users before you see them (Anthropic). Teams need both, and Part 13 covers the offline half.

What the Research Says About LLM Judges

The evidence for LLM judges is reasonable. Zheng and colleagues found that GPT-4 judges agreed with human preferences more than 80% of the time, matching the agreement between humans (Zheng et al.). The same paper documents position bias, verbosity bias, self-enhancement bias and limited reasoning on hard questions.

Position bias can be large. Wang and colleagues showed that reordering candidate answers was enough to make a weaker model appear to beat a stronger one on most test queries (Wang et al.). Their fixes are practical: ask the judge for evidence before a score, aggregate across orderings, and send the hardest cases to a person. For live evals on a single answer, the simplest protection is to grade each answer on its own against a clear rubric, with the explanation written before the verdict.

Which Evals Matter for a Data Agent

The evals worth running on a data agent map to the quiet failures above:

• Answer correctness: whether the answer addresses the question and is consistent with the SQL and the rows returned.

• Agent laziness: whether the agent stopped short of what was asked.

• Null or empty results: whether the query returned nothing useful.

• Chart failures: whether a requested chart failed to render or misrepresents the data.

• User frustration: whether the user repeated, rephrased or pushed back.

Some of these need no model at all. Anthropic's guide describes code-based graders as fast, cheap and reproducible, and model-based graders as flexible but non-deterministic, so detect empty results and chart errors with code and reserve the judge for questions of meaning.

The frustration eval has strong evidence behind it. Hassan and colleagues found that in search, a similar query issued within a short window, such as five minutes, implies dissatisfaction. A user who sends the same request to an agent twice in a row is giving the same signal.

Validating the Judges

A judge needs its own tests. Shankar and colleagues describe criteria drift, in which people refine their grading criteria as they read real outputs (Shankar et al.). Expect your rubric to change after you read the first batch of flagged conversations. Hamel Husain recommends binary pass or fail verdicts with a written critique, and measuring a judge's true positive and true negative rates separately against an expert's labels, because overall agreement can hide a judge that never flags anything (Husain).

Store each result against the span it judged. OpenTelemetry's gen_ai.evaluation.result event carries an evaluation name, a score value, a label such as pass or fail, and a free-text explanation, and should be parented to the operation being evaluated.

How Do You Turn an Eval Flag Into a Fix?

An eval flag is only useful if it leads to a change. Read the trace and decide which layer failed: the data, the context or the question. The flag type usually points to the right place.

• Empty or null results usually mean the agent filtered on a value it did not know, or the data changed under it. Check the column values and descriptions in the data model, and see Part 14 on data drift for keeping them current.

• Wrong answers with valid SQL usually mean a missing or ambiguous definition. Add the definition, then save a corrected version of the question as a golden question so the fix is tested from then on.

• Laziness, format problems and frustration usually mean an instruction gap. Adjust the system prompt or the relevant skill, which we cover in the agent harness guide.

• Unanswerable questions point to a gap in the data model. Either add the data or teach the agent to say clearly that the question is out of scope.

Then confirm the fix twice: watch whether the flag rate falls in the next conversations of the same type, and rerun your golden questions to check that nothing that used to work has broken. Our guide to self-improving agents explains why the quality of this feedback signal decides whether the loop actually improves the agent.

How Upsolve Approaches AI Agent Observability

In Upsolve, Observability lives in the agent chat history. Builders see every user, question, eval result and cost in one list. Opening a session shows the full trace: what was retrieved through RAG, which skill the agent used, the SQL it wrote, how SQL validation went and the chart it produced. Sessions can also be broken down into individual traces, each showing its spend, the number of charts generated and any errors.

Live LLM-as-judge evals run on real conversations as they happen, checking answer correctness, agent laziness, null or empty results, chart failures and user frustration. Embedded, Slack and Claude conversations all write to the same history, so a builder sees how the agent performs across every surface it is deployed on.

The demo uses a sample procurement project in which sample users such as Grace Tan and Carlos Ruiz have been running their own analyses. One sample conversation carries a user frustration flag. The user asked which suppliers make up the biggest share of spend, the agent answered in text, and the user, who wanted a bar chart to add to a canvas, sent exactly the same message again. As Serguei Balanovich, Upsolve, puts it in the demo: "The user sent the exact same message in a row and didn't get the bar chart they expected."

The fix took one edit. The builder opened System Prompts and added an instruction to always render charts when users ask analytical questions. The same workflow applies to other flags: change the prompt, skill or data model, then watch the next conversations. Good answers found in this history can be saved as golden questions, which is where Part 13 picks up.

Frequently Asked Questions

What is AI agent observability?

AI agent observability is the practice of recording each step an agent takes to answer a request and evaluating the quality of the result. A trace captures the context retrieved, the tools and skills used, any SQL generated, validation results, the output and the cost. Evals then score those traces for correctness, completeness and user experience. Together they show why an answer was wrong, which a latency or error dashboard cannot, so the cause can be fixed before users lose confidence.

How is agent observability different from LLM observability or APM?

Application performance monitoring tracks whether services are available and fast. LLM observability adds model calls, prompts, token counts and costs. Agent observability follows a multi-step run, including the tools chosen, the data each returned and how the answer was assembled, and attaches quality judgements to those steps. For data agents, this includes the SQL, the rows returned and the chart, because that is where many quiet failures begin.

What is LLM-as-a-judge, and can you trust it?

LLM-as-a-judge means using a language model to grade another model's output against a rubric. Research on MT-Bench and Chatbot Arena found that GPT-4 judges agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other. Judges also show position, verbosity and self-enhancement biases. They are trustworthy enough to run at scale when you give them binary criteria, ask for a written explanation and regularly compare their verdicts with an expert's labels.

Should you run evals on test sets or on production traffic?

Both, because they answer different questions. Offline evals on a fixed test set tell you whether a change broke anything before it ships, and they are repeatable. Live evals on production traffic show how the agent performs on the questions real users actually ask, including ones nobody anticipated. Failures found in production become new test cases, and test cases protect the fixes. Our guide to golden questions covers building that test set.

How can you detect user frustration with an AI agent?

Look at behaviour as well as explicit feedback, because unhappy users often just ask again. The strongest signal is repetition: the same or a very similar request sent again shortly after an answer. One Microsoft study of search found that a similar query within a short window, such as five minutes, implies dissatisfaction. Other signals include rephrasing, explicit corrections such as "no, I meant", requests for a different format and abandoned sessions. An LLM judge can read the conversation and label these patterns.

Does OpenTelemetry support AI agent tracing?

Yes, through its GenAI semantic conventions, which are still marked as in development. They define spans for model calls, retrievals, tool execution, agent invocation and skill loading. A conversation identifier links spans from the same session, and an evaluation event records a score, label and explanation against the operation it judged. Sensitive content such as prompts and outputs is opt-in, so teams choose what to capture.

Sources

1. OpenTelemetry: Semantic Conventions for GenAI agent and framework spans. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md

2. OpenTelemetry: Semantic Conventions for GenAI spans. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

3. OpenTelemetry: Semantic conventions for Generative AI events. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md

4. NeurIPS / Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

5. arXiv / Wang et al.: Large Language Models are not Fair Evaluators. https://arxiv.org/abs/2305.17926

6. arXiv / Shankar et al.: Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. https://arxiv.org/abs/2404.12272

7. Hamel Husain: Using LLM-as-a-Judge For Evaluation: A Complete Guide. https://hamel.dev/blog/posts/llm-judge/

8. Anthropic Engineering: Demystifying evals for AI agents. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

9. Microsoft Research / Hassan et al.: Beyond Clicks: Query Reformulation as a Predictor of Search Satisfaction (CIKM 2013). https://www.microsoft.com/en-us/research/publication/beyond-clicks-query-reformulation-as-a-predictor-of-search-satisfaction/

See every step your agent took, and fix what went wrong. Start free at upsolve.ai with 2,000 free AI credits, connect a data source and open the agent chat history to see your first traces and eval results. Prefer a walkthrough? Talk to our team. Read more about Upsolve Observability, and continue with Part 13: Golden Questions for AI Agent Evals.