The Complete Agent Stack · Part 11 of 14 · Distribution layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 10: Claude MCP for Data Analysis · Next: Part 12: AI Agent Observability →

Most AI data agent evaluations fail in one of two ways. Either the buyer watches a polished demo on curated questions, or the buyer opens an empty trial workspace and spends a week on setup before seeing a single answer. Neither tells you how the agent will behave on your data, for your users, after launch.

This guide gives you a checklist and a five-step method you can run in any product.

Key Takeaways

• Benchmark scores do not transfer to your schema: On Spider 2.0, built from real enterprise databases that often exceed 1,000 columns, o1-preview solved 21.3% of tasks, compared with 91.2% on the original Spider benchmark.

• One correct answer proves very little: Agents are non-deterministic, and Anthropic's guide to agent evals shows that an agent with a 75% per-trial success rate passes all three of three trials only about 42% of the time.

• Permissions must be tested as a control outside the model: The OWASP guidance on excessive agency is to implement authorisation in downstream systems instead of relying on the LLM to decide whether an action is allowed.

• A working sample project shortens the evaluation: When the connection, data model, security rules, agent and dashboards are already configured, you can run most of the checklist in your first session, before you connect your own data.

The 45-second version:

What Should You Evaluate in an AI Data Agent?

An AI data agent answers questions about structured data in natural language. It selects tables, writes and runs queries, and returns numbers, charts or a written explanation. What you are buying is a system of several parts, and the parts around the model decide whether its answers can be trusted.

A useful evaluation covers four layers:

• Ontology: the data model, metric definitions and table descriptions the agent reads, plus the row-level security that decides which rows each user can see.

• Agent harness: the skills, memory and tool use that turn a question into a correct query and a readable answer.

• Distribution: where answers reach people, such as inside your product, in a shared dashboard, in Slack or in Claude.

• Improvement loop: how failures are found, tested and fixed after launch.

A chat window only shows the output of these layers. A clean demo answer says nothing about whether the agent will respect permissions for your thousandth user, or whether anyone will notice when accuracy slips after a schema change.

Why Vendor Demos and Public Benchmarks Mislead Buyers

Text-to-SQL benchmarks are useful for comparing models, but they predict little about your warehouse, because scores depend heavily on how large and messy the databases are. BIRD was built to reflect real databases, with 12,751 question and SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains. Its authors emphasise dirty database values and the external knowledge needed to connect a question to the data, and they reported ChatGPT at 40.08% execution accuracy against a human result of 92.96%. Spider 2.0 moved closer still to enterprise work, and scores dropped again. We cover why this happens, and what metric definitions do about it, in our guide to the semantic layer for AI.

Demos have a different problem. A vendor chooses questions that the configured context handles well, so the demo measures preparation more than the product. An empty trial workspace fails the other way: it doesn't show what good looks like, so you cannot tell whether a poor answer reflects the product or your unfinished setup.

Two further properties rarely appear in demos. The first is consistency: the τ-bench authors found that even strong function-calling agents such as gpt-4o succeeded on fewer than 50% of tasks, and that pass^8 (succeeding on all eight repeated trials) fell below 25% in their retail domain. The second is ambiguity: the AMBROSIA benchmark tests questions with scope ambiguity, attachment ambiguity and vagueness, and found that even the most advanced models struggle to identify and interpret ambiguous questions. Business questions such as "who are our top customers?" are ambiguous by default, so test how the agent handles them.

The AI Data Agent Evaluation Checklist

Use this table as a scorecard. Run each test as at least two different users, and record the answer, the query and whether it matched your expected result.

Score the first eight rows on the agent's answers and the last four on the platform around it. Treat row-level security and consistency on decision-critical questions as pass or fail. The other rows can be weighted to suit your use case.

How to Run the Evaluation in Five Steps

Step 1: Start From a Working Reference

Begin with a project that is already configured end to end, such as a vendor's sample project. A working reference lets you test the agent, the security model and the distribution channels in your first session. It also shows what a correct configuration looks like, which helps later when your own setup misbehaves.

Time to first useful answer is itself a criterion. If a week of connecting, modelling and prompt writing comes before any answer, that cost will recur for every new dataset.

Step 2: Build a Small Golden Question Set

Write down 20 to 50 questions that real people ask, with the answers you expect. Anthropic's engineering team suggests that 20 to 50 simple tasks drawn from real failures is a great start, and recommends converting user-reported failures from your support queue into test cases. For a data agent, good sources are recent analyst requests, recurring dashboard questions and anything that has caused an argument about numbers.

Grade the result instead of the route the agent took. The same guide warns that checking for an exact sequence of steps produces brittle tests, because agents regularly find valid approaches the designer didn't anticipate. A different query that returns the correct figure should pass. Our guide to golden questions for AI agent evals covers how to maintain this set over time.

Step 3: Run Every Question Several Times

Ask each golden question at least three times in fresh sessions. Anthropic distinguishes pass@k, the chance of at least one correct answer in k attempts, from pass^k, the chance that all k attempts succeed. For an analytics agent, pass^k is the metric that matters, because a user who receives two different revenue figures for the same question will stop trusting both.

Read the transcripts of failed runs. They show whether the agent made a genuine mistake, misread an ambiguous question or was marked against an expected answer that was itself wrong.

Step 4: Attack the Permission Model

Sign in as, or impersonate, users from at least two tenants or regions and ask identical questions. Then try to talk the agent out of its rules with instructions to ignore restrictions, requests for "all customers" and questions that join to restricted tables.

The control you are looking for sits outside the model. OWASP recommends complete mediation, meaning authorisation is enforced in downstream systems, and recommends that actions taken for a user execute in that specific user's security context. If a vendor's answer to "how do you stop cross-tenant leaks?" relies on the system prompt, treat that as a failed test. Our guide to row-level security for AI agents explains the deterministic approach in detail.

Step 5: Connect Your Own Data and Compare

Once the platform passes on the reference project, connect a representative slice of your own data and re-run the golden set. Failures here usually point to context gaps such as missing column descriptions, undefined metrics or unclear joins. Comparing behaviour with the reference project separates platform problems from configuration problems.

This step also tests how much work stands between a trial and production. Our guide on how to set up a data agent walks through the setup steps you should expect.

How to Turn the Results Into a Decision

A scorecard is only useful if it separates problems you can fix from problems you cannot. Cross-tenant leaks, prompt-based permissions and inconsistent figures for core metrics are platform properties, and they should end an evaluation. Missing descriptions, undefined business terms and occasional chart errors are context gaps, so the question is how quickly the platform helps you close them.

Ask what happens to your golden set after purchase. In Anthropic's framing, capability evals target tasks the agent struggles with, while regression evals should pass at close to 100% and signal breakage when they slip. Your evaluation questions should become the regression suite that runs before every context change. A platform with no way to run that suite, review conversations or test changes in a draft leaves you without an improvement loop once real users arrive. Our guide to AI agent observability covers what to monitor in production.

How Upsolve Approaches Evaluation With Demo Projects

We built Demo Projects because the evaluation method above needs a working reference, and most buyers don't have one on day one. When you create a project in Upsolve, you can choose a sample project instead of an empty one. Upsolve sets up the connection, data model, security rules, agent, skills, memory, suggestions, charts and a dashboard for you, so you can learn how every Upsolve feature works on sample data before you commit to connecting your own.

In the founder demo, the sample project contains Olympics data plus geography and economic industry data. Querying as a sample Japan user, the first question is "What data do I have access to?", and the agent lists the Olympics, geography and industry data. A follow-up asking for the top athletes in summer 2004 returns a bar chart by gold medals and total medals. Because the sample row-level security is active, the Japan user only sees Japan's athletes. That covers the data awareness, accuracy and row-level security rows of the checklist within minutes.

The rest of the platform can be tested the same way:

• Distribution: copy the iframe into your product, or generate a preview link and use the agent as the Japan user would. In organisation settings, point the MCP agent at the sample project to ask the same governed questions from Claude, as described in our guide to data analysis in Claude with MCP.

• Charts and dashboards: favourite charts, build canvases and open the pre-built Olympics dashboard application, which shows medal trends over time and distribution by gender. Our guide to AI-generated dashboards covers the canvas side.

• Your own data: when you're ready, open project settings, go to Connections, pick your data source and wire it through the data model and agent.

The sample project stays useful after the evaluation. As Serguei Balanovich, Upsolve, put it in the demo: "You're able to cross-reference this and double-check that everything you've done in your actual production environment is correct." When your own project gives a strange answer, you can compare its data model, security rules and agent configuration against a workspace that's known to work.

Frequently Asked Questions

How do you evaluate an AI data agent?

Test it against questions with known answers, as more than one user, and more than once. Start from a fully configured project so you can test the agent, its security and its distribution channels together. Build 20 to 50 golden questions from real requests, run each several times and grade the result. Then try to break the permission model, connect a slice of your own data and re-run the set. Score accuracy, consistency, security, transparency, distribution and change control separately, and treat security failures as disqualifying.

How many test questions do you need to evaluate an AI analytics agent?

A small, well-chosen set is enough to start. Anthropic's engineering guidance suggests that 20 to 50 simple tasks drawn from real failures is a great start for agent evals. For a data agent, take questions from recent analyst requests, recurring dashboard questions and past disputes about numbers, and record the expected figure for each. Add ambiguous questions and questions over messy columns, because those expose weaknesses that clean questions hide. Keep the set as a regression suite after purchase.

What accuracy should you expect from an AI data agent?

There is no universal benchmark figure that predicts accuracy on your data. Public results vary widely with database complexity: on Spider 2.0, o1-preview solved 21.3% of enterprise tasks against 91.2% on the original Spider benchmark. The useful target is your own golden question set. Questions that feed decisions should pass consistently across repeated runs, and anything that is wrong should be traceable to a fixable cause, such as a missing definition, rather than random variation.

How do you test row-level security in an AI agent?

Ask the same questions as users from different tenants, regions or roles, and confirm that each user only sees their own rows every time. Then try to manipulate the agent with instructions to ignore its rules, requests for all records and queries that join to restricted tables. Finally, ask the vendor where the rules are enforced. OWASP recommends enforcing authorisation in downstream systems and executing actions in the specific user's security context, so a filter applied at query time is far stronger than an instruction in a prompt.

Should you evaluate an AI data agent on sample data or your own data?

Use both, in that order. A configured sample project lets you test the full platform, including security, dashboards and integrations, without days of setup, and it shows what a correct configuration looks like. Your own data then tests whether the agent copes with your schema, naming and messy values. Comparing the two helps separate platform weaknesses from gaps in your own context, such as missing column descriptions or undefined metrics.

What is pass^k and why does it matter for data agents?

Pass^k is the probability that an agent succeeds on all k independent attempts at the same task. It differs from pass@k, which measures the chance of at least one success. The τ-bench authors introduced pass^k to measure reliability and found that gpt-4o's pass^8 fell below 25% in their retail domain. For data agents, consistency matters because users compare answers. If the same revenue question returns two different figures, trust in every answer drops.

Sources

1. Lei et al., ICLR 2025: Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. https://arxiv.org/abs/2411.07763

2. Li et al., NeurIPS 2023: Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD). https://arxiv.org/abs/2305.03111

3. Yao, Shinn, Razavi and Narasimhan, Sierra: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. https://arxiv.org/abs/2406.12045

4. Saparina and Lapata, NeurIPS 2024 Datasets and Benchmarks Track: AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. https://proceedings.neurips.cc/paper_files/paper/2024/hash/a4c942a8405cc910f0a833d28d2573cc-Abstract-Datasets_and_Benchmarks_Track.html

5. Anthropic Engineering: Demystifying evals for AI agents. https://anthropic.com/engineering/demystifying-evals-for-ai-agents

6. OWASP Gen AI Security Project: LLM06:2025 Excessive Agency. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

To run this checklist on a working agent, start free at upsolve.ai with 2,000 free AI credits and create a sample project, or talk to our team about evaluating Upsolve on your own data.