When the database changes and the agent's description of it does not, answers degrade without any error message. This guide explains how to detect data drift in a data agent and how to repair it without touching production.
Ka Ling Wu
Co-Founder & CEO, Upsolve AI
10 min
The Complete Agent Stack · Part 14 of 14 · Improvement loop layer
This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.
← Previous: Part 13: Golden Questions for AI Agent Evals · Back to the start: Part 01: AI Agent Data Model
A data agent can be accurate at launch and wrong a few months later, even though nobody changed its model or its prompts. Tables are renamed, views stop refreshing, a column starts to mean something new, and the context the agent relies on quietly falls out of date.
Our guide to self-improving agents covers the general loop by which agents get better through context and memory. This one covers the specific failure that loop has to catch in analytics: drift between the database and the agent's picture of it.
Key Takeaways
Data dependencies are harder to track than code: Google engineers wrote in Hidden Technical Debt in Machine Learning Systems that data dependencies cost more than code dependencies, partly because no equivalent of compiler and linker analysis exists to detect them.
Small schema changes cause large accuracy drops: In the Dr.Spider benchmark, meaning-preserving changes to databases, such as renaming columns, cut accuracy for every model tested, and the most resilient model lost 50.7% on the hardest database perturbation.
Descriptions help, but ambiguous columns need people: In one study, LLM-generated column descriptions raised GPT-4o's zero-shot execution accuracy on the BIRD development set from 30.1% to 36.8%, and the authors recommend that domain experts describe inherently ambiguous columns.
Every drift fix needs a regression check: Anthropic's guide to agent evals says regression evals should pass at nearly 100%, so changes to a data agent's context belong in a draft that is tested before users see it.
The 45-second version:
What Is Data Drift in an AI Data Agent?
In machine learning, drift usually refers to statistical change. A widely cited review of concept drift defines it as unforeseeable changes in the underlying distribution of streaming data over time. That definition fits models trained on data whose inputs shift.
A data agent is exposed in a different way. It reads a description of the database, made up of the data model, table and column descriptions, metric definitions, prompts and knowledge files, and then writes queries against the live database. Data drift in a data agent is the gap that opens when the database changes and that description does not.
That gap takes four common forms.
Schema Drift
Tables are dropped or renamed, and columns are removed or change type. The dbt documentation on model contracts lists these as breaking changes for downstream consumers: removing a column, changing a column's data type, changing its constraints, and deleting, renaming or disabling a contracted model. A data agent is one of those consumers, and its data model may keep pointing at a table that no longer exists.
Freshness Drift
A view or source stops updating while its structure stays the same. Queries still succeed, so nothing fails visibly, but the agent reports last month's numbers as current.
Semantic Drift
A column keeps its name and type but changes meaning. New status values appear, a business rule changes, or an upstream team corrects a calculation. Sculley and colleagues describe how a silent correction to a mis-calibrated input can have sudden effects on a model that had fitted to the old values. Agents face the same risk when a definition they rely on changes without notice.
Coverage Drift
The business adds tables, products or regions that the context never described. Many tables ship without descriptions in the first place, so the agent has to guess what new columns mean from their names alone.
Why Data Drift Breaks Data Agents Quietly
The root cause is usually ownership. Sculley and colleagues describe how unstable data dependencies arise when the team that owns an input signal is separate from the team that consumes it, so updates to the input can be made at any time. Data agents often have the same split: a data team owns the warehouse, a product or analytics team owns the agent's context, and changes on one side are not always announced to the other.
Some drift is loud. A query against a deleted table returns an error, and someone notices. The harder cases return a plausible number. A stale view gives an outdated total, a renamed column leads the agent to a similar-looking column, and a changed definition produces a figure that is internally consistent but wrong.
Research on text-to-SQL shows how sensitive query generation is to these changes. Dr.Spider applied three kinds of meaning-preserving database perturbation: replacing column names with synonyms (country becomes nation), replacing them with abbreviations (ranking points becomes rank pts), and replacing a column with equivalent ones, such as splitting a full name into first and last names. Accuracy fell for every model on every database perturbation. The models in that study were earlier text-to-SQL systems, so the figures will differ for current LLMs, but the mechanism applies to any agent that maps words in a question to names in a schema.
Missing descriptions make the problem worse. The authors of the column description study note that relational databases often have uninformative descriptors, such as ambiguous columns and hard-to-interpret values. They also point out that all of the top 20 models on the BIRD benchmark used the benchmark's provided metadata to reach their scores. An agent with no descriptions has less to fall back on when names change. Our guide to the semantic layer for AI explains why that business context matters so much for accuracy.
How to Detect Data Drift Before Your Users Do
Detection works best as several overlapping checks, because each one catches a different form of drift.
Diff the Data Model Against the Live Schema
Compare every table and column the agent's data model references against the database's own catalogue, on a schedule and after each upstream deployment. Any reference that no longer resolves is schema drift. If your transformation layer supports schema contracts, such as dbt model contracts, breaking changes to contracted models can be caught before they ship.
Check Freshness on Sources and Views
Record when each source and view was last updated, and compare it against the freshness you expect. dbt's source freshness checks exist for this purpose, to show whether source data is meeting the service level agreement an organisation has defined. A stale view is usually a fix for the database side, and the agent's context should not paper over it.
Mine Conversations for Drift Symptoms
Chat history is the most direct record of drift as users experience it. Look for query errors, repeated rephrasing of the same question, users correcting the agent, and charts with broken date grouping. These signals are easy to collect and rarely read, because nobody has time to review every conversation. Our guide to AI agent observability covers how to capture them.
Run Regression Evals on Golden Questions
Keep a set of questions with known answers and run it after every change to the database or the context. Anthropic's guidance is that regression evals should pass at nearly 100%, and that a decline signals something is broken. For a data agent, a golden question whose answer changes without a business reason can be the first sign of semantic drift. Our guide to golden questions for AI agent evals explains how to build that set.
How to Repair Data Drift Without Breaking Production
Start by deciding where the fix belongs. A stale view or a missing table may need work in the database or the pipeline. A renamed column, a new table or a changed business rule needs an update to the agent's context: the data model, a description, a prompt rule, a knowledge file or a skill.
Generated descriptions are a reasonable starting point for coverage gaps, provided they are reviewed. The column description study proposes a two-step approach: generate descriptions automatically for columns with clear interpretation signals in the database, then have a domain expert describe the inherently ambiguous ones.
Context changes also need version control. Sculley and colleagues describe versioned copies of unstable inputs as a common mitigation, while noting that versioning brings its own costs, including potential staleness. For a data agent, the practical version is a draft workspace: make changes in a draft, run the golden set, compare results and promote a new version only when it passes. Our guide to versioning AI agent context covers that workflow.
The approaches below catch different kinds of drift, so they work best in combination.
Approach | What it catches | What it misses | Ongoing effort |
|---|---|---|---|
Manual data model upkeep | Changes the owner knows about | Changes made by other teams without notice | High and continuous |
Upstream schema contracts | Breaking changes to contracted models before they ship | Uncontracted tables, meaning changes and stale data | Moderate, set up per model |
Scheduled schema and freshness checks | Missing tables and columns, stale sources and views | Semantic drift where names and types stay the same | Low once automated |
Regression evals on golden questions | Changes that alter known answers | Questions outside the set | Moderate, grows with the set |
Agent-assisted review of usage | Errors and patterns in real conversations, undescribed tables | Problems users never hit; proposals still need human review | Low to run, review required |
How Upsolve Approaches Data Drift
The Builder Agent reads your workspace and your users' chat history, then proposes fixes to your data model, prompts and skills. It can do this because chat history, which is Upsolve's observability, sits in the same workspace as the context, the playground and the deployment. Every change lands in a draft, so production stays untouched until you promote it.
The founder demo runs on a sample procurement workspace with 12 tables. When asked to describe the tables, the Builder Agent found that only one, the contracts table, had descriptions, and that the purchase orders table had the wrong one. It ran database scans, then wrote table and column descriptions for every table that wasn't already described. The demo ran in auto mode, but each proposal can be approved one at a time instead. You can also upload files to the knowledge base, so the agent works from your real business rules as well as what it reads in the database.
The second request in the demo asked the Builder Agent to review the last month of usage and suggest data model improvements for performance and accuracy. It scanned the chat history and found:
Four tables in the data model that no longer exist in the database.
A monthly view that had gone stale, which the demo flagged as a fix for the database side.
Formatting issues and a chart date error. The presenter asked it to add a date cast rule, and it added date truncation and timestamp-with-time-zone rules to the chart generation prompt.
The missing tables are a textbook case of schema drift. As Serguei Balanovich, Upsolve, put it in the demo: "The database has been changed under the hood and no one came in and modified the data model to account for that change. That is what we call data drift, and it kills these sorts of data agents if it's left unchecked."
Beyond descriptions and prompts, the Builder Agent can write knowledge base files and generate skills from what it learns about your data. Each change appears in the draft's history. You test the result in the playground, then promote the draft to a new version for your users. That keeps automated drift repair inside the same review and versioning process as any other change.
Frequently Asked Questions
What is data drift in AI agents?
For a data agent, data drift is the gap that opens when the underlying database changes and the agent's context does not. The context includes the data model, table and column descriptions, metric definitions, prompts and knowledge files. Common causes are dropped or renamed tables, stale views, columns whose meaning changes and new tables nobody has described. The model itself is unchanged, so the agent keeps answering confidently, but its queries are based on an outdated picture of the data. Some drift causes visible errors, while the rest produces plausible numbers that are wrong.
What is the difference between data drift and schema drift?
In machine learning, data drift usually means a change in the statistical distribution of a model's inputs, often discussed as concept drift. Schema drift is a structural change: tables or columns are added, removed, renamed or retyped. For a data agent, schema drift is the most common and most detectable form of drift, because you can diff the agent's data model against the live database. Semantic drift is harder to spot, because a column keeps its name and type while its meaning or values change.
How do you detect data drift in a data agent?
Combine four checks. Diff the tables and columns in the agent's data model against the database catalogue on a schedule and after upstream deployments. Monitor freshness on the sources and views the agent queries. Review chat history for query errors, rephrased questions, user corrections and chart errors. Run a golden question set with known answers after every change, and investigate any answer that moves without a business reason. Each check catches drift the others miss, which is why teams use several together.
Can an AI agent write its own table and column descriptions?
Yes, and research suggests it helps. In a study using the BIRD development set, LLM-generated column descriptions raised GPT-4o's zero-shot execution accuracy from 30.1% to 36.8%. The same study found that models struggle with inherently ambiguous columns, and recommends generating descriptions automatically where the data gives clear signals, then having a domain expert describe the ambiguous ones. In practice, generated descriptions should go into a draft for review, alongside any business documents that explain what the data means.
How often should you check a data agent for data drift?
Tie the checks to the events that cause drift. Diff the data model against the live schema after every upstream deployment or migration, and on a regular schedule in case changes go unannounced. Run freshness checks at the cadence your sources are meant to update. Review usage on a fixed cycle; the Builder Agent demo, for example, reviewed the last month of conversations. Run the golden question set before every change to the agent's context is promoted. Our guide to self-improving agents covers how these signals feed a wider improvement loop.
Should an agent be allowed to fix data drift automatically?
An agent can do most of the detection and drafting, because it can read the schema and the chat history faster than a person. Applying changes straight to production is riskier, because a wrong description or prompt rule affects every user at once. A safer pattern is to let the agent propose or apply changes in a draft, run the golden question set against that draft, and have a builder promote it. Some fixes, such as refreshing a stale view, belong in the database and need a data engineer.
Sources
Sculley et al., Google, NeurIPS 2015: Hidden Technical Debt in Machine Learning Systems. https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
Lu, Liu, Dong, Gu, Gama and Zhang, IEEE Transactions on Knowledge and Data Engineering: Learning under Concept Drift: A Review. https://arxiv.org/abs/2004.05785
Chang et al., ICLR 2023: Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness. https://arxiv.org/abs/2301.08881
Wretblad et al., Table Representation Learning Workshop at NeurIPS 2024: Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance. https://arxiv.org/abs/2408.04691
dbt Developer Hub: Model contracts. https://docs.getdbt.com/docs/collaborate/govern/model-contracts
dbt Developer Hub: Source freshness. https://docs.getdbt.com/docs/deploy/source-freshness
Anthropic Engineering: Demystifying evals for AI agents. https://anthropic.com/engineering/demystifying-evals-for-ai-agents
To keep your own data agent in step with your database, start free at upsolve.ai with 2,000 free AI credits, or talk to our team about running the Builder Agent on your workspace.
Get a context layer and a working agent in minutes
Connect your warehouse, encode what your metrics actually mean, and launch an analytics agent your team can trust.
Every answer cited
30+ data connectors
Deploy to Slack & Teams
