The Complete Agent Stack · Part 13 of 14 · Improvement loop layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 12: AI Agent Observability · Next: Part 14: Data Drift in AI Agents →

Part 13 covers fixed reference answers and how to decide whether an answer is confident enough to show. Its companion, Part 12 on AI agent observability, covers the production side: tracing every step and running LLM-as-judge evals on live conversations.

Key Takeaways

• A small set drawn from real failures is enough to start. Anthropic's guide to agent evals suggests 20 to 50 simple tasks taken from real failures, and defines a good task as one where two domain experts would independently reach the same verdict (Anthropic).

• Grade the result, not the SQL text. Zhong and colleagues found that the Spider benchmark's metric at the time marked correct queries as wrong 2.5% of the time on average and 8.1% in the worst case, and proposed comparing query results across a distilled suite of test databases instead (Zhong et al., EMNLP 2020).

• Vague business terms need a written answer. On the AMBROSIA benchmark, which covers scope ambiguity, attachment ambiguity and vagueness, even the most advanced models struggled to recognise and interpret ambiguous questions, and the ambiguity remained when the database was provided (Saparina and Lapata, NeurIPS 2024).

• A model's own confidence is a weak gate. Xiong and colleagues found that LLMs tend to be overconfident when they state their confidence, so a useful confidence score should draw on evidence from the run itself, such as whether a trusted reference matched (Xiong et al., ICLR 2024).

The 45-second version:

What Are Golden Questions?

A golden question is a natural-language question paired with a reference answer your team has checked. For a data agent, the reference is usually the SQL query that produces the right result, sometimes with the expected chart or output format. A collection of them is a golden dataset: the agent's equivalent of a regression test suite.

Golden questions do two jobs. As test cases, they check that the agent still answers known questions correctly after a prompt, skill, data model or model change. As references, they give the agent a worked example to follow when a user asks the same or a similar question. Other analytics tools use the same pattern: Snowflake's Cortex Analyst keeps a Verified Query Repository of questions and SQL, and uses a matching verified query to generate its answer when a user asks something similar (Snowflake).

Golden questions sit on top of the data model and the metric definitions covered in our semantic layer guide. Definitions say what a term means in general, and a golden question shows exactly how that meaning becomes a query for a specific request.

Why Does an Agent Answer the Same Question Differently?

Three causes explain much of the inconsistency in data agents.

The first is non-determinism. The same input can produce different outputs, so one correct run proves little. Our guide to evaluating an AI data agent explains why the pass^k metric, which requires every one of k attempts to succeed, matters more for analytics than a single success.

The second is ambiguity. Many business questions have more than one reasonable reading. "Who are our worst vendors?" could rank by spend, by order count, by late deliveries or by defect rate. The AMBROSIA benchmark counts vagueness as one of three types of ambiguity, and found that even the most advanced models struggle to identify and interpret ambiguous questions whose ambiguity persists with the database in view (Saparina and Lapata). A model that does not notice the ambiguity will pick an interpretation, and it may pick a different one next time.

The third is that good answers disappear. When someone confirms an answer was right, that judgement usually stays in chat history, and the next user gets a fresh attempt. Golden questions turn those confirmations into context and tests, the kind of reliable feedback signal our guide to self-improving agents argues every improvement loop needs.

How Do You Build a Golden Dataset From Real Conversations?

The best golden questions come from questions users have already asked, because those reflect real vocabulary and real priorities. Synthetic questions written by the builder can match the schema too neatly.

Start From Production Traces

Use your observability history as the source. Look at the most frequent questions, the ones that feed decisions and the ones your live evals flagged. Anthropic's guide recommends turning user-reported failures into test cases.

Review the Question, the SQL and the Result

Someone who knows the data should open each candidate conversation and check what the user meant, what SQL the agent wrote, and whether the result and chart are right. If the answer is correct, save it as it is.

Edit the SQL to Encode the Business Rule

If the answer is close but wrong, correct the SQL before saving it. The corrected query then records a business rule that no schema contains. If your company ranks vendors by spend, the golden version of "worst vendors" should order by spend and nothing else.

Prioritise Ambiguous Terms

Cover the terms that have caused disagreement first: contested metrics, vague superlatives such as "best" and "worst", and date ranges such as "last quarter". If two people who know the business would disagree on whether an answer passes, write down the rule before you save the question.

Version the Set With the Rest of the Context

Golden SQL is written against a specific data model and set of definitions, so it may need updating when either changes. Keeping it in the same release as prompts and definitions, as described in Part 04 on context versioning, means you always know which set of tests applies to which version.

How Do You Run Golden Questions as Test Cases?

Run the set whenever something changes: a prompt edit, a new skill, a data model update or a new model version. Anthropic distinguishes capability evals, which measure what an agent can newly do and start with a low pass rate, from regression evals, which should hold a pass rate near 100% and signal breakage when they slip. A golden dataset is a regression suite, so any drop is worth investigating.

Compare Results, Not Query Text

Two correct SQL queries can differ in aliases, join order or subqueries, so grading by text match fails valid answers. Executing both queries and comparing results is closer to what users care about, but a single database can let a wrong query pass when its result happens to match on today's data. Zhong and colleagues addressed this by running queries against a small suite of generated databases chosen for high coverage of the reference query, which gives a tighter estimate of whether the two queries mean the same thing (Zhong et al.).

Test Paraphrases as Well as the Saved Wording

If golden questions also serve as references, a test that asks the exact saved question mostly checks that retrieval found it. Add a few paraphrases of each important question, such as "which suppliers cost us the most" alongside "worst vendors", so the test checks that the agent applies the rule to new wording.

Grade What the Agent Produced

Anthropic's guide warns against grading the exact sequence of tool calls, because agents find valid paths the designer did not expect, so grade the result, the chart and any stated assumption. Where the output is text or a chart, an LLM judge can compare it with the reference answer, with the same calibration checks described in Part 12.

What Is a Confidence Score, and Where Should You Set the Threshold?

Tests run before a change ships. A confidence score runs on every production answer, estimates how likely it is to be right and, with a threshold, decides whether it reaches the user.

Why Self-Reported Confidence Falls Short

Asking the model how sure it is seems obvious, but the evidence suggests caution. Xiong and colleagues found that LLMs tend to be overconfident when they verbalise confidence, and that every method they tested struggled on tasks requiring professional knowledge (Xiong et al.). Kadavath and colleagues found that larger models are well calibrated on multiple-choice questions in the right format, but that their estimates of whether they know an answer were harder to calibrate on new tasks (Kadavath et al.). Questions about a company's private warehouse are, by definition, new tasks for any model.

The same Kadavath study found a more useful pattern: a model's estimate that it knows the answer rose appropriately when relevant source material was in its context. Evidence about what context the agent had, and how well it fit the question, is a better basis for confidence than the model's opinion of itself.

Evidence-Based Confidence Signals

A practical confidence score for a data agent combines signals that can be observed in the run:

• Reference match: whether the question matched a golden question, and how trusted that golden question is.

• Schema fit: whether the tables and columns the agent fetched match what the user asked about, or the agent had to reason its way to an answer.

• Output quality: whether the query produced a usable result and, where one was expected, a good chart.

Some analytics tools already expose a version of this. Snowflake's documentation, for example, notes that its API response includes a confidence field showing which verified query was used.

Choosing the Threshold

Every threshold trades coverage for accuracy. Research on selective classification formalised this as the risk-coverage trade-off: a model that may decline some inputs can guarantee a lower error rate on the ones it answers. Geifman and El-Yaniv showed an image classifier that guaranteed 2% top-5 error on ImageNet with 99.9% probability while answering almost 60% of test images (Geifman and El-Yaniv). A survey of abstention in LLMs notes that declining to answer is increasingly recognised for its potential to reduce hallucinations (Wen et al.).

The right threshold depends on who reads the answer. An analyst exploring data can check a weak answer, while a customer reading an embedded report usually cannot. A sensible sequence is to log scores with the threshold off, compare score bands with eval outcomes from production, and then raise the threshold for the audiences where a wrong number costs the most.

How Upsolve Approaches Golden Questions and Confidence Scores

In Upsolve, Golden Questions is a repository of questions your agent should answer consistently correctly, saved with the SQL you trust. The agent uses them as references for repeatable questions, and builders use them as test cases to check that the agent still gets them right after every change.

Golden questions come from real conversations. A builder opens a past chat in observability, reviews the question, the SQL and the chart, and saves it, editing the SQL first if needed.

The demo uses a sample procurement project. One past conversation asked for total spend and PO count by month, and its trace showed SQL generation, data retrieval and chart generation. The answer looked right to someone familiar with the data, so it was saved as a golden question unchanged. Another asked for the worst vendors, and the agent's SQL ordered vendors by both total spend and total orders. As Serguei Balanovich, Upsolve, explains in the demo: "Worst vendors can be kind of vague, but in my platform, worst vendors is about spend." The builder removed total orders from the ordering, kept only total spend, and saved the corrected query, so future test runs check that the agent ranks vendors by spend.

Golden questions also feed a confidence score on every answer. The score is based on whether the answer matched golden questions and how correct they were, whether the schemas and columns fetched fit the question, and whether a good chart was produced. All answers go through by default. Sensitive teams, or teams unsure whether an agent is ready for end users, can set a high threshold so that low-confidence answers are held back and users see only high-quality results.

Frequently Asked Questions

What is a golden dataset for AI agents?

A golden dataset is a set of questions with reference answers that a team has checked and agreed are correct. For a data agent, each item is usually a question paired with trusted SQL. The dataset works as a regression suite: you run it after every change to the agent's prompts, skills, data model or model, and investigate any question that stops passing. It can also give the agent worked examples to follow when similar questions arrive.

How many golden questions do you need?

You need fewer than many teams expect. Anthropic's guidance on agent evals suggests that 20 to 50 simple tasks drawn from real failures is a good start. Begin with the questions that feed decisions, the terms that cause disagreement and the failures your production evals flagged, then add questions as new failures appear. A small set that experts agree on beats a large set nobody has checked.

How are golden questions different from few-shot examples?

They can be the same artefact used for two purposes. A few-shot example is shown to the model to guide its answer, while a test case is held out to check the answer. When a golden question serves as both, a test on its exact wording mainly checks that the agent found the reference. Add paraphrases of important questions to the test set, so you also check that the agent applies the business rule to wording it has not seen.

How do you test a text-to-SQL agent?

Run a set of questions with trusted reference queries, execute both the agent's SQL and the reference SQL, and compare the results. Avoid grading by SQL text, because correct queries can be written in many ways. Be aware that results can match by coincidence on a single database, which is why research on text-to-SQL evaluation compares queries across a suite of test databases. Run each question several times, since one correct run does not show that the agent is consistent.

What is a confidence score for an AI agent's answer?

A confidence score estimates how likely a specific answer is to be correct. Asking the model to rate itself is unreliable, because research shows LLMs tend to be overconfident when they state their confidence. Stronger scores combine evidence from the run, such as whether the question matched a trusted golden question, whether the tables and columns retrieved fit the question, and whether the query produced a usable result and chart. The score can then gate which answers reach users.

Should low-confidence answers be blocked from users?

It depends on the audience. Blocking answers below a threshold lowers the error rate of what users see but means some questions go unanswered, which research on selective prediction calls the risk-coverage trade-off. For internal exploration, letting every answer through and showing the evidence is often fine. For customer-facing reports or regulated teams, a higher threshold is safer. Start with the threshold off, compare scores with real outcomes, and raise it where a wrong number costs most.

Sources

1. Anthropic Engineering: Demystifying evals for AI agents. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

2. EMNLP / Zhong, Yu and Klein: Semantic Evaluation for Text-to-SQL with Distilled Test Suites. https://arxiv.org/abs/2010.02840

3. NeurIPS / Saparina and Lapata: AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. https://arxiv.org/abs/2406.19073

4. ICLR / Xiong et al.: Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. https://arxiv.org/abs/2306.13063

5. arXiv / Kadavath et al.: Language Models (Mostly) Know What They Know. https://arxiv.org/abs/2207.05221

6. arXiv / Geifman and El-Yaniv: Selective Classification for Deep Neural Networks. https://arxiv.org/abs/1705.08500

7. TACL / Wen et al.: Know Your Limits: A Survey of Abstention in Large Language Models. https://arxiv.org/abs/2407.18418

8. Snowflake Documentation: Cortex Analyst Verified Query Repository. https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst/verified-query-repository

Teach your agent the right answer, and know when it's unsure. Start free at upsolve.ai with 2,000 free AI credits, save your first golden question from a real conversation, and see how the agent's confidence changes. Prefer a walkthrough? Talk to our team. Read more about Upsolve Golden Questions, and continue with Part 14: Data Drift in AI Agents.