The Complete Agent Stack · Part 05 of 14 · Ontology layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 04: AI Agent Context Versioning · Next: Part 06: AI Agent Harness →

This guide closes the ontology layer of the series. Our earlier guide on building a data agent that works in production covers the strategic questions, including build versus buy. This one is the hands-on version. The steps apply whichever tool you use, and we use Upsolve and its setup agent, Dash, as the worked example near the end.

Key Takeaways

• Raw tables are not enough. On the BIRD benchmark of 95 real databases, the best model tested at publication, ChatGPT, reached 40.08% execution accuracy against 92.96% for humans (Li et al.), and the authors name dirty values and missing external knowledge among the causes.

• Let a model draft descriptions, then review the ambiguous ones. Wretblad et al. found that LLM-generated column descriptions consistently improved text-to-SQL performance, while models struggled with inherently ambiguous columns that still needed expert input.

• Give the agent read access only. OWASP's guidance on excessive agency recommends limiting an agent's permissions "to the minimum necessary", for example read access to the tables it needs and no ability to insert, update or delete.

• Split domain procedures into skills. In Anthropic's Agent Skills design, each skill costs about 100 tokens of metadata until it is triggered and under 5,000 tokens when loaded (Claude docs), so focused skills keep detailed procedures out of context until a question needs them.

The 45-second version:

What Is a Data Agent, and What Does Setup Involve?

A data agent is an AI system that takes a question in plain language, finds the relevant tables and definitions, writes and runs a query, and returns an answer, usually with a chart and an explanation. Our guide to what a data agent is covers the concept in more depth.

Setup is everything around the model: a connection to the data, a data model that describes the tables, files that hold business rules, skills for recurring analyses, instructions for the agent, a round of testing, and a published version that users can reach.

Anthropic's guidance on agents applies directly. It recommends "finding the simplest solution possible, and only increasing complexity when needed" (Anthropic, Building effective agents). For a data agent, the simplest version that works is a small, well-described set of tables and a short list of questions it must answer correctly.

What Do You Need Before You Start?

Most teams already have these five things in some form.

A list of real questions

Collect 20 to 50 questions people actually ask the data team, with the answers you trust. These become your first test set and, later, your golden questions.

Read-only access to the data

You need a database role or service account that can read the relevant tables and nothing else. Request it early so setup is not left waiting on an access approval.

The documents that hold your business rules

Gather metric definitions, a glossary, notes on known data problems, and any internal pages that explain how numbers are calculated. These often live in a wiki, a CRM or a set of Markdown files.

A reviewer who knows the data

Name one analyst or domain expert who can confirm whether a description or an answer is right. Without that person, nobody can say when the agent is ready.

A decision about where answers will appear

Answers might appear in an internal chat tool, a Slack channel, an embedded panel in your product or an AI assistant such as Claude. The destination affects permissions and testing, so decide before you publish.

How to Build a Data Agent in Eight Steps

Step 1: Pick one domain and write down what it must answer

Start with one area, such as procurement spend or subscription revenue, rather than the whole warehouse. Narrow scope keeps the context small and makes testing possible. Anthropic's context engineering guidance describes the goal as "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome" (Anthropic).

Step 2: Connect your data with least privilege

Connect the warehouse, database or source systems with the read-only credentials from your preparation. If your data sits across several SaaS tools without a warehouse, choose a setup that can query them together with SQL, which Part 03 on MCP versus SQL compares in detail. If you are evaluating a tool before connecting anything sensitive, start with its sample data.

Step 3: Choose the tables and describe them

Expose only the tables the first domain needs. Then give every table and column a description that says what a row represents, what each value means and which columns are safe to filter or join on.

This is the step most worth automating. Wretblad et al. found that generated descriptions improved accuracy, especially with larger models, but that "models unsurprisingly struggle with columns that exhibit inherent ambiguity, highlighting the need for manual expert input" (Wretblad et al.). A sensible split is to let a model draft every description and have your reviewer check the ambiguous ones: status codes, flags, amounts that may or may not include tax, and dates with more than one meaning. Part 01 on the AI agent data model covers what a complete model contains.

Step 4: Write the business rules down as context files

Turn the documents from your preparation into short files the agent can read: how to count a customer, which table is the source of truth, which periods have known data problems, and which assumptions to state in an answer. BIRD's authors identify "external knowledge between NL questions and database contents" as one of the core challenges for text-to-SQL (Li et al.), and context files are where that knowledge lives.

If the rules already sit in tools like Notion or HubSpot, connecting them through the Model Context Protocol, "an open-source standard for connecting AI applications to external systems", lets the setup read them directly instead of copying them by hand.

Step 5: Package recurring analyses as skills

A skill is a packaged procedure for one type of question, such as a vendor spend breakdown or a cohort retention calculation. Skills keep the system prompt short, because the agent loads a skill's full instructions only when a question needs them. In Anthropic's design, only a skill's name and description sit in context until it is triggered (Claude docs). If your data covers two unrelated domains, give each its own skill.

Step 6: Write the agent instructions

The system prompt should explain who the users are, how to write SQL against this data, how to choose and check charts, and when to ask a clarifying question instead of guessing. Anthropic's context engineering guide advises writing prompts "at the right altitude for the agent", which means clear heuristics rather than brittle, hard-coded rules. Keep domain detail in the context files and skills, and keep the prompt about behaviour.

Step 7: Test as real users and read the trace

Run your question list against the agent while signed in as several representative users, not only as an administrator, because permissions change what each person can see. For each answer, read the trace: which tables were selected, which skill ran, what SQL was written and whether the chart was checked. Anthropic's guide to building agents recommends "extensive testing in sandboxed environments, along with the appropriate guardrails" for autonomous agents. A draft or playground environment is that sandbox.

Step 8: Publish a version and deploy where people already work

When the agent answers the test set correctly, publish the configuration as a numbered version so you can compare and roll back later. Part 04 on AI agent context versioning explains why the whole bundle should ship as one release. Then deploy to the destination you chose, invite a small group of users and watch their conversations. Every question the agent handles badly points to a missing description, rule or skill.

Setting Up by Hand vs With a Setup Agent

The three common ways to produce a first draft differ mainly in who writes the descriptions, rules and skills.

The setup agent approach changes where human time goes. People stop writing boilerplate descriptions and spend that time reviewing ambiguous columns and business rules, which is the work only they can do.

Common Data Agent Setup Mistakes

Exposing every table at once

Wide scope adds noise to every query and makes testing much harder. Add tables when a real question needs them.

Accepting generated descriptions without review

Generated descriptions are a strong first draft. Ambiguous columns are where they go wrong, and those columns often hold the amounts and statuses people care about most.

Putting everything in one long prompt

A single prompt that holds every rule and procedure is hard to test and expensive to load. Move procedures into skills and facts into context files.

Testing only as an administrator

An administrator sees all rows, so a release can pass every test and still fail for the users it was built for. Test with restricted users, and confirm that row-level security holds.

Editing the live agent directly

Changes made in production cannot be tested before users see them and cannot be cleanly rolled back. Work in a draft and publish versions.

How Upsolve Approaches Data Agent Setup

Upsolve's self-serve onboarding covers the same ground, with Dash, Upsolve's setup agent, drafting the data model descriptions, context files, skills and instructions from steps 3 to 6. Anyone can sign up on the free tier, which includes 2,000 one-time AI credits and all pro features, limited only by credits.

After creating a project, you connect data. Options include any of 300 source systems and APIs through the Data Plane, a direct database connection, a CSV upload or the sample data. When you launch a workspace, Dash asks whether to connect MCP servers, such as Notion or HubSpot, that hold business context, and whether to upload files such as Markdown documents with business logic rules. It then scans every table, column, MCP server and file, which takes from one to 30 minutes depending on how much there is to read. Dash uses the same logic as the Builder Agent, applied to a workspace that starts from zero.

In the demo, run on a sample connection that holds both Olympic data and business financial data, setup took about three and a half minutes. Dash wrote five table definitions and 26 column definitions. It added context files explaining how to read the table values, how to count Olympic medals, and the assumptions and caveats it found while scanning. Because the data covered two unrelated domains, it created two skills, an Olympic medal analysis skill and an NZ business financials skill, and wrote the agent instructions.

All of this landed in a draft beside an empty version one, so it could be reviewed before anyone used it. After publishing the draft as version two, the demo opened the playground and asked what data was available, then which countries had won the most Olympic gold medals. The trace showed each step. Chart generation was slower than it should be, which the team noted as something for the Builder Agent to improve. As Serguei Balanovich, Upsolve, explained in the demo, Dash "helps you set up your workspace from zero and get started on the right foot, so you don't have to waste time on the initial boilerplate setup."

From there, the chart can be favourited and added to a canvas, which offers a snapshot from when the query ran and a live view that updates with the data. The workspace can then be embedded in a product or shared by inviting the team. If you want to explore before connecting anything, Part 11 on evaluating a data agent covers the pre-built demo projects, and the Self-Serve Onboarding feature page has the full overview.

Frequently Asked Questions

What is a data agent?

A data agent is an AI system that answers questions about your data in plain language. It reads a description of your tables and business rules, writes and runs a query, and returns an answer, often with a chart and an explanation of how it was calculated. Unlike a dashboard, it handles questions nobody anticipated. That flexibility is why the setup work matters: the agent can only be as accurate as the descriptions, rules and examples it receives.

How long does it take to build a data agent?

The connection itself can take minutes. Most of the time goes into writing table descriptions, business rules, skills and instructions, which teams doing it by hand often spend weeks on before the agent is reliable. A setup agent can draft that context in minutes; in Upsolve, Dash takes from one to 30 minutes depending on the volume of data, MCP servers and files. Budget separate time for a domain expert to review the draft and run your test questions.

Do I need a data warehouse to build a data agent?

Not necessarily. The agent needs tables it can query with SQL and a description of what they mean. A warehouse is the most common home for those tables, but a direct database connection, uploaded CSV files or a data plane that connects SaaS tools into one data model can also work. The important choice is to query real tables with real SQL, so answers can be verified, rather than stitching together results from separate API calls.

Can AI write the table and column descriptions for me?

Yes, and it is usually the right place to start. Research on BIRD-based datasets found that LLM-generated column descriptions consistently improved text-to-SQL accuracy. The same research found that models struggle with inherently ambiguous columns, so the generated draft needs review. Ask your reviewer to focus on status fields, flags, amounts that may include or exclude tax or discounts, and any column whose name means different things to different teams.

Should I start with sample data or my own data?

Start with sample data if you are learning how a tool works, since it shows the full setup without access requests. Move to your own data as soon as you want to know whether the agent is useful, because only your schema, your business rules and your users' questions will test that. Many teams do both: a short session on sample data, then a narrow first domain on real data.

How do I know when my data agent is ready for users?

Run your list of real questions against the agent as several representative users and compare each answer with the one your team trusts. It is ready for a small group when it answers the agreed set correctly and says so clearly when it cannot answer. After launch, review conversations weekly, add the questions it got wrong to your test set and publish fixes as new versions.

Sources

1. Li et al. (NeurIPS 2023): Can LLM Already Serve as a Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD). https://arxiv.org/abs/2305.03111

2. Wretblad et al.: Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance. https://arxiv.org/abs/2408.04691

3. OWASP GenAI Security Project: LLM06:2025 Excessive Agency. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

4. Anthropic (Claude Platform Docs): Agent Skills Overview. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview

5. Anthropic Engineering: Building Effective Agents. https://www.anthropic.com/engineering/building-effective-agents

6. Anthropic Engineering: Effective Context Engineering for AI Agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

7. Model Context Protocol: What Is the Model Context Protocol (MCP)? https://modelcontextprotocol.io/docs/getting-started/intro

Try Upsolve

Start free at upsolve.ai with 2,000 free AI credits and let Dash build your first workspace, or talk to our team about your data. Next, Part 06 covers the AI agent harness that runs on top of this context.