Data Agent Architecture: The 14 Components a Production Data Agent Needs

A production data agent is a stack of four layers, from the data model it reads to the loop that keeps it accurate, and most failures after launch trace back to a component missing from one of them.

Key Takeaways

• Context moves accuracy more than model choice. On the BIRD text-to-SQL benchmark, GPT-4's execution accuracy rose from 34.88% to 54.89% when each question came with a short piece of external knowledge about the data (Li et al., NeurIPS 2023).

• Benchmark scores overstate readiness. On Spider 2.0, built from real enterprise databases that often exceed 1,000 columns, o1-preview solved 21.3% of tasks, compared with 91.2% on the original Spider benchmark (Lei et al., ICLR 2025).

• Permissions cannot live in the prompt. The OWASP Top 10 for LLM Applications says the system prompt should not "be used as a security control" and that authorisation checks "must not be delegated to the LLM" (OWASP LLM07).

• Accuracy has to be measured repeatedly. Agents are non-deterministic, so an agent that succeeds on 75% of single attempts passes all three of three attempts only about 42% of the time (Anthropic).

What Is a Data Agent?

A data agent is an AI system that answers questions about an organisation's data by planning an analysis, writing and running queries against real databases, checking the results and returning an answer, usually with a chart. It differs from a chatbot in that it acts on live data, and from a single text-to-SQL call in that it takes several steps, uses tools and can recover when a step fails.

A production data agent is one that people outside the data team rely on without checking every answer. That bar changes what the system needs. The agent has to use the company's own definitions, respect who may see which rows, work where people already ask questions, and stay accurate as the data changes underneath it.

Why Do Data Agents Fail in Production?

Most data agents perform well in a demo and degrade in use. The model rarely changes in that time. The schema, the business definitions and the people asking questions do, and an agent that lacks a component to absorb those changes starts giving confident wrong answers.

Three causes account for most of the gap.

• Missing business context. Raw tables do not say which of several revenue definitions finance uses. On BIRD's 95 real databases, the best model tested at publication reached 40.08% execution accuracy against 92.96% for humans, and the authors name dirty values and missing external knowledge among the reasons (Li et al.).

• Controls placed in the wrong layer. Instructions in a prompt are suggestions to a model. The UK National Cyber Security Centre argues that prompt injection "may never be totally mitigated" because a language model draws no line between data and instructions (NCSC).

• Silent drift. Data dependencies are harder to track than code dependencies (Sculley et al.), and in the Dr.Spider benchmark, meaning-preserving database changes such as renamed columns cut accuracy for every model tested (Chang et al., ICLR 2023). Nothing errors. The answers simply get worse.

Each cause maps to a layer that is absent or incomplete. The rest of this guide describes those layers and the components inside them.

The Complete Agent Stack at a Glance

The Complete Agent Stack groups the 14 components into four layers. Each layer depends on the one below it, and the top layer feeds back into the bottom one.

1. Ontology: the data described clearly, with permissions enforced outside the agent.

2. Agent harness: the software that turns that context into answers and learns from each user.

3. Distribution: the places answers reach people, from dashboards to the assistants they already use.

4. Improvement loop: live evaluation that finds the gaps, so the context can be hardened to close them.

The table lists every component, the failure it prevents and a quick way to test for it.

Layer 1: Ontology

The ontology is everything the agent knows about the data before it writes a query: what the tables mean, how they join, which values exist, which rows each person may see and how all of that is versioned.

Data model and metric definitions

An agent reads a registered description of tables, columns, keys, live column values and metric definitions before it writes SQL. Without one, it guesses which of several "active customer" definitions applies, and different users get different numbers. Descriptions are measurable context: in one study on BIRD, GPT-4o's execution accuracy rose from 30.13% to 36.78% when columns had generated descriptions (Wretblad et al.). Read the full guide: Part 01, AI Agent Data Model.

Row-level security

Permissions for a data agent belong in the query path, applied by the server before any query runs, so that no wording of a question can widen what a user sees. This matters most for multi-tenant products, where one agent serves every customer. A well-designed rule fails closed: if no rule matches, no rows return. Read the full guide: Part 02, Row-Level Security for AI Agents.

Cross-source SQL

Business questions often span tools such as billing, product analytics and a CRM. Fetching each source through separate tool calls routes every intermediate result through the model, which is slow and error-prone. Anthropic's engineers describe one workflow that fell from 150,000 tokens to 2,000 when the data was handled in code instead (Anthropic). A query layer that joins sources with real SQL keeps the model out of the arithmetic. Read the full guide: Part 03, MCP vs SQL for Data Agents.

Context versioning

An agent's context lives in several places: the data model, the knowledge base, skills, examples and prompts. Small edits to any of them are production changes. In one study, meaning-preserving formatting changes moved a model's accuracy by up to 76 points (Sclar et al.). Versioning the whole bundle as one release lets a team test a draft and roll back if it misbehaves. Read the full guide: Part 04, AI Agent Context Versioning.

Setup from raw tables

The first trusted answer depends on how quickly a team can describe its data. A model can draft table and column descriptions, but ambiguous columns still need a person who knows the business (Wretblad et al.). The agent should also start with read-only access to the tables it needs, in line with OWASP's advice to limit an agent's permissions "to the minimum necessary" (OWASP LLM06). Read the full guide: Part 05, How to Set Up a Data Agent.

Layer 2: Agent Harness

The harness is the software around the model. It decides what the model sees, which tools it can call, which checks run before an answer is shown and what the agent remembers about each user.

Harness, skills and memory

LangChain defines an agent harness as "every piece of code, configuration, and execution logic that isn't the model itself" (LangChain). For a data agent, a good harness picks the right tool for each step, runs and checks the SQL, and recovers when the data surprises it. That recovery needs an external signal: an ICLR 2024 study found that models struggle to correct their own reasoning without feedback (Huang et al.). Skills hold a team's analysis methods, and per-user memory can shape presentation without changing the numbers. Read the full guide: Part 06, AI Agent Harness.

MCP connections

The Model Context Protocol lets an agent reach the tools a team already uses, to pull context from documents or to act on an answer, for example by saving an analysis or opening a ticket. Each connection adds risk as well as reach. Anthropic's engineers note that "more tools don't always lead to better outcomes" (Anthropic), so allowlists and per-connection instructions matter, and a person should be able to block any tool. Read the full guide: Part 07, MCP Servers for Data Agents.

Layer 3: Distribution

Distribution decides whether an accurate answer reaches the people who need it. A correct chart that stays in one person's chat history has little value to anyone else.

Dashboards from chat

Useful answers get lost in conversation threads. In a Nielsen Norman Group study of people using ChatGPT, participants kept going back to earlier answers to build their next prompt, a behaviour the researchers called "apple picking" (NN/g). Saving a chart with the conversation behind it, then arranging charts on a shared canvas, turns one-off answers into a dashboard a team can maintain and question. Read the full guide: Part 08, AI-Generated Dashboards.

Branded reports

Teams often rebuild dashboards by hand in slides before they share them. An AI report builder can restyle a dashboard as a branded app, but the output still has to meet accessibility rules, such as WCAG 2.2's 4.5:1 contrast ratio for normal text (W3C). Public links should carry snapshots, since shareable URLs leak through browser history and referrer headers (W3C TAG). Read the full guide: Part 09, AI Report Builder.

Governed access in AI assistants

Many people now research and draft in a general-purpose assistant, and their data questions follow them there. Connecting an assistant to a governed data agent keeps the same definitions and permissions in that new channel. The key detail is per-user sign-in: Claude's connector documentation says tools that act as the individual user should use OAuth, not a shared API key (Claude Docs). Read the full guide: Part 10, Claude MCP for Data Analysis.

Sample projects for evaluation

Buyers and new users judge a data agent from their first session, and an empty workspace shows nothing about how the layers work together. A fully configured sample project lets someone test accuracy, permissions and sharing before connecting their own data. Because public benchmarks transfer poorly to real schemas (see Spider 2.0 above), the evaluation should then repeat on a slice of the buyer's own data. Read the full guide: Part 11, How to Evaluate an AI Data Agent.

Layer 4: Improvement Loop

The improvement loop finds where the agent is wrong in real use and feeds fixes back into the ontology. Without it, a data agent is only as good as it was on launch day.

Observability and live evals

Observability means recording every step the agent took, including the SQL, the tool calls and the cost, and scoring live conversations automatically. LLM-as-judge evaluation is practical at scale: in the MT-Bench study, strong model judges matched human preferences more than 80% of the time, the same level as agreement between people (Zheng et al.). Judges still need checking against human labels, because they carry measurable biases. Read the full guide: Part 12, AI Agent Observability.

Golden questions

Golden questions are the questions an agent must answer correctly, saved with the SQL the team trusts. They work as regression tests and as references the agent can learn from. Anthropic suggests starting with 20 to 50 tasks drawn from real failures (Anthropic). Grading should compare query results, since judging SQL text alone marked correct queries as wrong in up to 8.1% of cases in one study (Zhong et al.). Read the full guide: Part 13, Golden Questions for AI Agent Evals.

Drift repair

Databases change after launch. Tables are renamed, views are dropped and new columns appear, while the agent's description of the data stays the same. In Dr.Spider, the most resilient model lost 50.7% accuracy on the hardest database perturbation (Chang et al.). Repairing drift means comparing the data model with the live database and real usage, proposing fixes in a draft, and rerunning the golden questions before anything reaches users. Read the full guide: Part 14, Data Drift in AI Agents.

Is Your Data Agent Production-Ready? A Self-Assessment

Answer each question with yes or no. Every "no" points to the component, and the guide, to work on next.

1. Does every core metric have one written definition that the agent reads before it writes SQL?

2. Are row-level permissions enforced by the server, outside the prompt, and tested as more than one user?

3. Can the agent answer a question that needs two data sources without stitching results together in its context?

4. Can you change a prompt, skill or definition in a draft, test it and roll it back?

5. Could a new team connect its data and get a correct answer in the first session?

6. Does the agent run and check its SQL before it answers, and recover when a query fails?

7. Are external tools limited to an allowlist, with a way for a person to block any call?

8. Can a good answer be saved, shared and questioned by a teammate without a screenshot?

9. Do people reach the same governed agent from the tools they already use, under their own identity?

10. Can you see every step of yesterday's failed conversations, with automatic scores on live traffic?

11. Do you have 20 or more golden questions that rerun after every change?

12. Will you know within a day if a schema change breaks the agent's picture of the data?

How Upsolve Approaches the Agent Stack

We built Upsolve around this stack so that a team does not have to assemble the 14 components from separate tools. Each component above is a feature with its own guide in this series, and they share one versioned workspace: the data model, knowledge base, skills, golden questions, prompts and memory are edited together and shipped as one release.

Check out Upsolve's The Complete Agent Stack: 45-Second Tour playlist for a quick walkthrough.

The ontology layer covers data models with versioned metric definitions, row-level security enforced server side before any query runs, a data plane that writes real SQL across sources such as Stripe, PostHog and HubSpot without a warehouse, and workspaces that version everything together. Dash, Upsolve's setup agent, reads a new team's data, MCP servers and files and builds the first workspace as a draft for review.

The rebuilt harness picks the right tool for each step, checks its work and remembers each user's preferences. Answers then reach people through BI canvases, branded HTML apps, an embedded iframe, Slack or Claude. Observability, golden questions and the Builder Agent close the loop, and every change the Builder Agent proposes lands in a draft first. A sample project sets up the whole stack on sample data, so a team can explore it before connecting its own.

Frequently Asked Questions

What is a data agent?

A data agent is an AI system that answers questions about an organisation's data. It plans an analysis, writes and runs queries against real databases, checks the results and returns an answer, often as a chart. Unlike a chatbot, it works on live data, and unlike a single text-to-SQL call, it takes several steps and can recover when one fails. In production, it also needs the company's definitions, permission rules and a way to measure its own accuracy.

What is the difference between a data agent and text-to-SQL?

Text-to-SQL converts one question into one query. A data agent wraps that step in a harness that chooses tools, runs the query, checks the result, retries on errors and presents the answer. It also reads context that a bare text-to-SQL model lacks, such as metric definitions and column values, which is why context can raise accuracy more than a model upgrade on benchmarks like BIRD.

What does a data agent need before it goes into production?

It needs a described data model with metric definitions, permissions enforced outside the model, a versioned way to change its context, a harness that checks its own SQL, and channels where people already work. After launch, it needs observability, golden questions and drift repair. The self-assessment above turns these into twelve yes-or-no checks.

Can a data agent enforce data permissions?

It should not be the thing that enforces them. Security guidance such as the OWASP Top 10 for LLM Applications says authorisation must not be delegated to the model. Row-level rules should apply in the server or database before any query runs, based on the signed-in user's properties, so that no prompt can widen access. The agent then sees only the rows that user is allowed to see.

How do you measure a data agent's accuracy?

Combine fixed tests with live evaluation. Golden questions with trusted SQL work as regression tests that rerun after every change, graded on query results and not on SQL text. Live LLM-as-judge evals score real conversations for correctness, empty results and signs of frustration. Because agents are non-deterministic, each test should run more than once.

Do you need a data warehouse to run a data agent?

Not necessarily. A data agent needs a query layer it can run SQL against, with permissions applied. That can be a warehouse, or a layer that connects operational sources such as billing, product analytics and CRM tools into one model. What matters is that joins happen in SQL, outside the model's context, and that the agent reads one data model across every source.

The Full Series

Each guide covers one component in depth. Read them in order, or start with the layer you are working on.

Ontology

1. AI Agent Data Model: How to Give Every Answer One Definition

2. Row-Level Security for AI Agents: Permissions a Prompt Can't Override

3. MCP vs SQL for Data Agents: Why Cross-Source Analytics Needs Real Joins

4. AI Agent Context Versioning: How to Ship an Agent as One Release

5. How to Set Up a Data Agent: Eight Steps From Raw Tables to Trusted Answers

Agent harness

6. AI Agent Harness: How Skills and Memory Make Data Agents Reliable

7. MCP Servers for Data Agents: How to Add Context and Actions Safely

Distribution

8. AI-Generated Dashboards: How to Turn Chat Answers Into Shared BI

9. AI Report Builder: How to Turn Dashboards Into Branded Data Apps

10. Claude MCP for Data Analysis: How to Get Governed Answers in Claude

11. How to Evaluate an AI Data Agent: A Buyer's Checklist Before You Commit

Improvement loop

12. AI Agent Observability: How to Catch Wrong Answers in Production

13. Golden Questions for AI Agent Evals: Test Cases and Confidence Scores

14. Data Drift in AI Agents: Why Data Agents Decay After Launch

Build the whole stack in one workspace. Start free at upsolve.ai with 2,000 free AI credits and every feature in this series, or talk to our team about deploying a data agent in your product.

Sources

1. Li et al., NeurIPS 2023: Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD). https://arxiv.org/abs/2305.03111

2. Lei et al., ICLR 2025: Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. https://arxiv.org/abs/2411.07763

3. OWASP Gen AI Security Project: LLM07:2025 System Prompt Leakage. https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/

4. OWASP Gen AI Security Project: LLM06:2025 Excessive Agency. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

5. Anthropic Engineering: Demystifying evals for AI agents. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

6. UK National Cyber Security Centre: Prompt injection is not SQL injection (it may be worse). https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection

7. Sculley et al., Google, NeurIPS 2015: Hidden Technical Debt in Machine Learning Systems. https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf

8. Chang et al., ICLR 2023: Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness. https://arxiv.org/abs/2301.08881

9. Wretblad et al.: Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance. https://arxiv.org/abs/2408.04691

10. Anthropic Engineering (Jones and Kelly): Code execution with MCP: Building more efficient agents. https://www.anthropic.com/engineering/code-execution-with-mcp

11. Sclar, Choi, Tsvetkov and Suhr, ICLR 2024: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. https://arxiv.org/abs/2310.11324

12. LangChain: The Anatomy of an Agent Harness. https://www.langchain.com/blog/the-anatomy-of-an-agent-harness

13. Huang et al., ICLR 2024: Large Language Models Cannot Self-Correct Reasoning Yet. https://arxiv.org/abs/2310.01798

14. Anthropic Engineering (Aizawa): Writing effective tools for agents, with agents. https://www.anthropic.com/engineering/writing-tools-for-agents

15. Nielsen Norman Group: Accordion Editing and Apple Picking: Early Generative-AI User Behaviors. https://www.nngroup.com/articles/accordion-editing-apple-picking/

16. W3C Web Accessibility Initiative: Understanding Success Criterion 1.4.3, Contrast (Minimum), WCAG 2.2. https://www.w3.org/WAI/WCAG22/Understanding/contrast-minimum.html

17. W3C Technical Architecture Group: Good Practices for Capability URLs. https://www.w3.org/2001/tag/doc/capability-urls/

18. Claude Docs: Authentication for connectors. https://claude.com/docs/connectors/building/authentication

19. Zheng et al., NeurIPS 2023: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

20. Zhong, Yu and Klein, EMNLP 2020: Semantic Evaluation for Text-to-SQL with Distilled Test Suites. https://arxiv.org/abs/2010.02840