To build a data agent that works in production, you need four things: a governed warehouse, a semantic layer, encoded institutional knowledge, and an evaluation suite that tests accuracy before real users ever see an answer.

Most teams get the model right and skip the context, which is exactly why the demo dazzles and the deployment stalls. This guide walks through what to build, in what order, and the mistakes that quietly send data agents back to the lab.

If you have already tried this once and watched it fall apart with real questions, you are not alone, and the problem is almost certainly not your model. It is the layer underneath it.

The Production Gap: Why Most Data Agents Never Ship

Here is the pattern almost every analytics engineering lead recognizes. You wire up a large language model to your warehouse, feed it a schema, and ask it a question. It writes SQL, returns a chart, and the room goes quiet in a good way. Then you hand it to a business user, who asks "how did revenue trend last quarter," and the agent confidently returns a number that is wrong, because it summed bookings instead of recognized revenue and included a test account nobody told it to exclude.

That gap between the demo and the deployment is where the money goes to die, and the research is blunt about the scale of it. MIT's State of AI in Business 2025 found that 95% of enterprise generative AI pilots deliver no measurable business impact. A closer look at the drop-off explains why: plenty of organizations evaluate these tools, only about a fifth reach a pilot, and just 5% make it to production. The bottleneck is not the model. It is that static systems cannot learn from feedback, adapt to context, or improve once real work starts flowing through them.

The agentic wave has its own version of this warning. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, pointing to runaway costs, fuzzy business value, and weak risk controls. It also flags how much of the market is agent washing: chatbots, robotic process automation, and assistants relabeled as agents without the autonomy to back the name. A data agent that only answers questions you anticipated is a dressed-up dashboard. A data agent that handles the questions you did not anticipate, correctly, is a different thing to build.

The good news is that the failure modes are well understood, which means they are avoidable. Building a data agent that survives contact with real users is less about model selection and more about the infrastructure you put underneath it.

What You Actually Need to Build a Data Agent

Before any prompt engineering, get these four foundations in place. Think of them less like a checklist and more like load-bearing walls: the agent is the roof, and it collapses without all four. They map directly onto the three-layer context architecture that determines whether an agent can be trusted, plus the warehouse those layers sit on.

A Governed Data Warehouse (The Structure Layer)

The agent needs a single, reliable place to query, and it needs to understand what lives there. A warehouse like Snowflake, BigQuery, Databricks, or Redshift provides the Structure layer: the schemas, tables, columns, lineage, and relationships that tell the agent what data exists and how it connects.

This is table stakes, and it is also where a surprising number of builds fail before they start. If your pipelines are fragmented, your table names are cryptic, and half your logic lives in a reporting layer nobody documented, the agent inherits all of that ambiguity. Clean, well-modeled data is not a nice-to-have here. It is the floor.

A Semantic Layer (The Meaning Layer)

Structure tells the agent what a revenue column is. It does not tell the agent what "revenue" means at your company. Is it ARR? Run rate? Recognized revenue in the current fiscal quarter? Does it exclude churned accounts, refunds, or that one enterprise contract finance always treats separately?

A semantic layer encodes those definitions so the agent stops guessing. It is the difference between an agent that returns a number and an agent that returns the right number. If you are setting this up for the first time, our guide on what a semantic layer is and how to structure one walks through the fundamentals.

One caveat worth stating plainly: a semantic layer is necessary, but it is not sufficient on its own. It handles Meaning. It does not handle Trust, and it does not capture the unstructured tribal knowledge that never made it into a metrics definition. That gap is the next foundation.

Encoded Institutional Knowledge (The Trust Layer)

This is the layer almost everyone skips, and it is the one that decides whether your agent is production-ready or perpetually POC-ready. It is the context that a thoughtful analyst at your company carries in their head: which table is actually authoritative when two conflict, why Q3 numbers look strange because of a data migration, that "active user" is defined differently for the finance team than the product team, and which past answers have already been verified as correct.

For the past decade, that context has lived in Slack threads, in people's memory, and in a dbt YAML that was last touched in 2021. AI tools have no access to it, so they fail, every time. Encoding it is the single highest-leverage move in the entire build. This is the heart of the discipline that context engineering for analytics exists to formalize: what context an agent needs, how to curate it, and how to keep it current.

An Evaluation Suite (How You Prove Trust)

You cannot ship what you cannot measure. Before a data agent touches real users, you need a way to test its answers against known-correct results, catch regressions when you change something, and quantify accuracy, hallucination rate, latency, and cost.

The core mechanism is golden query testing: a curated set of questions with verified answers that act as unit tests for your agent. Run the suite on every change. When accuracy on a golden query drops, you know before your users do. We go deep on this in our guide to evaluation frameworks for production agents, including how to build the datasets and which metrics actually predict production reliability.

How to Build a Data Agent: Step by Step

With the foundations understood, here is the build sequence. The order matters. Teams that jump to step five without steps one through four are the ones filling out the 40% cancellation statistic.

Step 1: Map Your Data and Access Model

Start by inventorying what the agent will be allowed to see and do. Which sources? Which tables are authoritative? What is the access control model, so the agent only surfaces data a given user is authorized to see?

This is where you also decide scope. Resist the urge to point the agent at everything. Pick the domain where the queue hurts most (often the same handful of repeat questions your team answers weekly) and map that first. In many organizations, a large share of the analytics request queue is repeat questions (Upsolve customer data puts it around 47%), which means a narrow first target still covers a wide swath of demand.

Step 2: Encode the Semantic and Business Context

Now translate institutional knowledge into structured context the agent can use. Metric definitions, KPI logic, business rules, join paths, and the tribal knowledge that qualifies them. If you have a dbt or Cube semantic model, this feeds the Meaning layer directly. If you do not, you encode definitions from scratch.

Pro tip: Do not treat this as a one-time data dump. The context you encode on day one will have gaps you cannot see yet. Design for a loop where real user conversations surface missing context, a human reviews it, and the context store improves. That compounding is what separates an agent that is 70% accurate at launch from one that is 95% accurate three months later.

Step 3: Constrain the Agent With Validated Patterns and Guardrails

An unconstrained agent generating novel SQL against a complex warehouse is a liability. The teams that reach production do the opposite of "let the model figure it out." They encode validated SQL patterns for common question types, set behavioral guardrails and scope rules, and give the agent a library of trusted building blocks rather than a blank slate.

This is also where you decide what the agent should refuse. A production-grade agent that says "I do not have a verified definition for that metric, here is who to ask" builds more trust than one that confidently invents an answer. The ability to hold back is a capability, not a gap.

Step 4: Build Your Evaluation Suite Before You Ship

Assemble your golden query set: the real questions users ask, paired with answers you have verified as correct. Aim for coverage across your priority domain, including the awkward edge cases that broke your last attempt. Then wire the suite to run automatically so every context change or prompt change is scored against it.

Define your bar for production. Not perfection, but a threshold you can defend: accuracy on the golden set, an acceptable hallucination rate, latency users will tolerate, and a cost per query the business can absorb. Ship when you clear the bar, not when the demo looks good.

Step 5: Deploy Narrow, Then Expand

Launch to a small group in the surface where they already work: Slack, Teams, an embedded component in your product, or a web interface. Watch the conversation logs. Every question the agent handled poorly is a context gap, and every gap you close makes the next hundred questions better.

Expand scope only after the narrow deployment holds. This is the compounding improvement loop in action, and it is why the fifth step never really ends. A data agent is a system you tune, not a project you finish.

Where Data Agent Builds Break Down

Most failed builds fail for the same handful of reasons. If you recognize your last attempt here, that is useful, because each of these is fixable.

Treating Accuracy as a Model Problem

The most expensive assumption is that a better or bigger model will fix accuracy. It will not. In a16z's analysis of why data agents stall, the recurring culprit is tooling that cannot hold onto feedback, adjust to context, or get better over time. The failure is context, not the model. Swapping models buys you marginal gains on a problem that lives one layer down.

Shipping Without an Evaluation Suite

Teams under pressure to ship often treat evaluation as a phase-two luxury. Then they ship, an executive catches a wrong answer, trust evaporates, and the project is quietly shelved. Without golden query testing, you have no way to know if a change made the agent better or worse, and no way to prove reliability to the stakeholders whose confidence you need.

Trying to Cover Everything at Once

Pointing the agent at every table and every question at once guarantees a mediocre experience everywhere and a great one nowhere. Narrow scope is not a limitation; it is the strategy. A data agent that nails the top ten repeat questions is more valuable, and far more trusted, than one that attempts everything and gets a third of it wrong.

Stopping at the Semantic Layer

A well-maintained semantic layer is a genuine advantage, and it is also incomplete on its own. It covers Meaning. It does not encode the Trust layer (which answers are verified, which corrections users have made) or the unstructured tribal knowledge that never fit into a metric definition. Teams that stop at the semantic layer build agents that are consistent and confidently wrong on the questions that live in the gaps.

A Realistic Timeline: From Zero to Production

How long this takes depends almost entirely on one choice: whether you build the context infrastructure yourself or start from something that already handles it.

A full in-house build realistically runs 4 to 6 months. That covers modeling and cleaning the warehouse, standing up or extending a semantic layer, building the context encoding workflow, writing the evaluation harness, and the inevitable iteration once real users find the edges. None of that is wasted work; it is the work. It is just a lot of it, and it competes with everything else on your roadmap.

The alternative is to build on infrastructure that already solves the context and evaluation layers. With a platform purpose-built for this, teams are frequently production-ready in about 14 days (Upsolve customer data), because the encode-deploy-tune loop, the guardrails, and the golden query testing come as a foundation rather than something you assemble from parts.

Neither path is automatically right. Which one fits depends on your team's capacity, your appetite for maintaining context infrastructure long term, and how urgent the queue problem is. That tradeoff is worth working through in detail, and it is worth mapping against your team's capacity and timeline before you commit to either path.

Build vs. Buy: Which Path Fits Your Team

A quick way to pressure-test the decision. Building in-house makes sense when analytics infrastructure is a core competency you want to own, you have engineering capacity to spare (rare), and your requirements are unusual enough that no platform fits. Buying, or building on top of a context infrastructure platform, makes sense when time-to-value matters, when maintaining a context and evaluation layer is not where you want your best engineers spending their quarters, and when you have already tried the in-house route and hit the wall.

The MIT data offers a sobering data point for the build-everything instinct: externally partnered AI deployments succeeded roughly twice as often as internal builds in their sample. That is not a mandate to buy. It is a reminder that the operational discipline (context retention, evaluation, continuous improvement) is the hard part, and it is worth being clear-eyed about whether you want to build that discipline from scratch.

If you decide the context infrastructure is not what you want to hand-roll, a purpose-built platform can serve as that foundation while your team spends its time on the domain knowledge only you have. Either way, the four foundations and the five build steps are the same. The only question is how much of it you assemble yourself.

Start With the Layer Most Teams Skip

Building a data agent that survives real users comes down to the layer most teams skip: the encoded institutional knowledge that makes answers trustworthy. Get the warehouse, the semantic layer, the context, and the evaluation suite in place, and the model almost takes care of itself. If you would rather not assemble that infrastructure from scratch, see how platforms built for context engineering handle the Structure, Meaning, and Trust layers so your team can spend its time on the domain knowledge only you have.

Frequently Asked Questions

What is a data agent?

A data agent is an AI system that receives a question in plain language, retrieves the relevant context (schema, metric definitions, business rules), generates and runs an analysis, and returns a validated answer. Unlike a copilot that assists a human writing queries, a production data agent acts autonomously, which is why the context and evaluation layers underneath it matter so much.

How long does it take to build a data agent?

A full in-house build typically takes 4 to 6 months, covering warehouse preparation, semantic modeling, context encoding, and evaluation. Using a platform that provides the context infrastructure, teams are often production-ready in about 14 days (Upsolve customer data). The variable is almost entirely how much of the context and evaluation layer you build yourself.

Why do most data agents fail in production?

They fail because of context, not the model. Independent research on enterprise AI has found that the large majority of pilots never deliver measurable business impact, largely because the systems cannot retain context, learn from feedback, or handle questions no one anticipated. An agent that only answers pre-planned questions is a dashboard in disguise.

Do I need a semantic layer to build a data agent?

A semantic layer helps a great deal because it encodes the Meaning layer (what your metrics actually mean), but it is not sufficient on its own. You also need the Structure layer (a governed warehouse) and the Trust layer (verified answers and encoded tribal knowledge). Some platforms can encode context without a pre-existing semantic model, which lowers the barrier for teams that never built one.

How do I know if my data agent is accurate enough to ship?

Build a golden query suite: real user questions paired with verified-correct answers. Define a production bar across accuracy, hallucination rate, latency, and cost per query, and ship only when the agent clears it on that suite. Do not use "the demo looked good" as your threshold; demos test the questions you chose, not the ones users will ask.

Should I build a data agent or buy one?

Build if analytics infrastructure is a core competency, you have spare engineering capacity, and your needs are genuinely unusual. Buy, or build on a platform, if time-to-value matters, you do not want to maintain context infrastructure long term, or a previous in-house attempt already stalled. Industry research shows partnered AI deployments tend to reach production more often than internal builds, largely because the operational discipline is the hard part.