The Complete Agent Stack · Part 04 of 14 · Ontology layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 03: MCP vs SQL for Data Agents · Next: Part 05: How to Set Up a Data Agent →

Part 04 sits in the ontology layer. It deals with a problem that appears as soon as more than one person edits an agent: the context that drives it lives in several places, changes on different schedules, and cannot be shipped or rolled back as a whole.

Key Takeaways

• Small context edits are production changes. In a study of prompt formatting, Sclar et al. measured accuracy differences of up to 76 points on LLaMA-2-13B from meaning-preserving format changes alone, so the only safe way to judge an edit is to run it.

• Agent context is entangled. Google's paper on hidden technical debt in machine learning named the principle "Changing Anything Changes Everything", and the same holds when a renamed metric, a new skill and an old prompt rule meet in one agent.

• The model moves too. Chen, Zaharia and Zou found GPT-4's accuracy on identifying prime numbers fell from 84% in March 2023 to 51% in June 2023, which is why a release record should name the model and why tests must rerun when it changes.

• Release the whole bundle together. Google's SRE book describes a branch-and-cherry-pick release method under which "we know the exact contents of each release" (Google SRE book), and an agent needs the same property across its data model, knowledge base, skills, examples and prompts.

The 45-second version:

What Is AI Agent Context Versioning?

AI agent context versioning is the practice of capturing everything that shapes an agent's answers as one numbered, immutable release. That includes the data model and metric definitions, knowledge base documents, skills, example questions with trusted SQL, and the system prompt. A release can be tested, deployed to a surface and rolled back as a unit.

Agent config management is the wider discipline around it: who can change each piece, where drafts live, how changes are tested, and which release each surface runs.

Anthropic describes context engineering as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference", and lists system instructions, tools, MCP, external data and message history as parts of that context (Anthropic). Versioning is the release-management half of the same discipline. Once context has been curated, the team still needs to know which curated context produced a given answer. Our context engineering guide covers the curation half.

Analytics agents raise the stakes because the data underneath them also changes. As we argued in why Git-style versioning breaks for data analytics agents, you cannot version the warehouse itself. Versioning the context does not freeze the data, but it removes one large source of unexplained change, so that when an answer shifts you can tell whether the context or the data moved.

Why Does Agent Context Drift Apart?

In most teams each piece of an agent has its own home. The data model sits in dbt or YAML files, the system prompt sits in application code, skills sit in a repository, golden questions sit in a spreadsheet, and business rules sit in a wiki. Each has an edit history, but none of them records which combination was live when a customer saw a wrong number.

The first reason this matters is entanglement. Sculley and colleagues at Google wrote that in machine learning systems "No inputs are ever really independent", and called this the CACE principle: Changing Anything Changes Everything (Sculley et al., NeurIPS 2015). In an agent, renaming a metric in the data model changes which example questions match a user's request. A new skill can overlap with an older instruction in the system prompt, and the agent may follow either one depending on how a question is phrased.

The second reason is sensitivity. Sclar et al. showed that formatting choices which do not change meaning can move accuracy by up to 76 points, and that the sensitivity remained with larger models and more examples (Sclar et al.). Reading a diff therefore tells you very little about what an edit will do. You have to run the changed agent against real questions.

The third reason is volume. The same Google paper observed that in a mature system "the number of lines of configuration can far exceed the number of lines of the traditional code. Each configuration line has a potential for mistakes." Agent context fits that description: hundreds of table and column descriptions, dozens of rules and examples, and several skills, all edited by different people.

The fourth reason is that the model is a moving part. Chen, Zaharia and Zou concluded that "the behavior of the 'same' LLM service can change substantially in a relatively short amount of time" (Chen et al.). If the release record does not name the model, a regression caused by a provider update looks identical to one caused by a prompt edit.

Together these produce the symptom most teams recognise. Debugging a bad answer means jumping between chat logs, traces and configuration to reconstruct what the agent saw, and changing one piece means guessing how it affects the rest.

What Belongs in a Versioned Agent Release?

If changing something can change an answer, it belongs in the release, or the release must record which version of it was used.

Data model and metric definitions

This covers the tables the agent may query, the joins between them, the table and column descriptions, and the governed metric definitions. Most other pieces refer to it, so a change here can affect all of them. Part 01 on the AI agent data model covers what it should contain.

Knowledge base and context files

This covers business rules, glossaries, caveats about known data problems, and instructions such as how to count a customer. These documents fall out of line with the data model when they are edited separately, which is the strongest argument for shipping them together.

Skills

Skills are packaged procedures for recurring analyses, such as a spend analysis or a cohort calculation. A changed skill is a changed behaviour, even when no prompt was touched.

Golden questions and example assets

Questions with trusted SQL and approved charts act as retrieval examples and as test cases. Versioning them with the data model keeps the tests consistent with the definitions they test. Part 13 on golden questions explains how to build the set.

System prompts and agent instructions

These are the instructions for writing SQL, choosing and validating charts, and handling ambiguity. Anthropic recommends prompts written "at the right altitude for the agent", specific enough to guide behaviour without hard-coding brittle logic, which makes them easier to test as part of a release.

What stays outside the release

Per-user memory and chat history are runtime state. They accumulate between releases and should not roll back when the context does. They still belong next to the release, because they are the evidence you need when reviewing what a release did in practice. Our guide to AI agent memory covers how to manage them. Access rules such as row-level security are usually governed separately, but every release should be tested against them.

Per-Artefact Versioning vs Whole-Context Releases

Many teams start by versioning each artefact on its own. In the MLflow Prompt Registry, "Once created, a prompt version cannot be modified", and aliases such as production or staging point at a specific version, so rollback means moving the alias (MLflow docs). A dbt project in Git gives the data model a similar history.

The gap is between artefacts. Google's SRE book notes that "configuration changes are a potential source of instability", and its release process exists so the team knows exactly what each release contains (Google SRE book). Thoughtworks' Continuous Delivery for Machine Learning makes the same point for models, asking that code, data and models move "in small and safe increments that can be reproduced and reliably released at any time" (Martin Fowler / CD4ML).

The whole-context approach costs more testing per release. In return, every answer maps to one release, and a rollback restores a combination known to work together.

How to Test and Promote a Context Change

A release process for agent context needs only a few rules.

Work in a draft, never in production

Every edit, whether written by a person or proposed by an agent, should land in a draft that end users cannot reach. Production keeps serving the last approved release until someone promotes the draft.

Test as real users, not as an administrator

An administrator usually sees every row. A tenant user, a regional manager or a new hire sees less, and the same question can return a different answer or nothing at all. Test the draft while signed in as representative users, including one who has never used the agent and has no stored memory.

Read the trace step by step

A final answer can look right for the wrong reason. Check which examples were retrieved, which tables and columns were selected, which skill ran and whether the chart passed validation. A trace turns "it got worse" into a specific step that changed.

Rerun the golden questions against both versions

Run the full set of golden questions against the draft and against production, then compare the SQL as well as the results. Any question that changes should be explained before promotion.

Promote, then watch

After promotion, monitor live conversations for regressions, since real users will ask questions the test set missed. Chen et al.'s call for "continuous monitoring" applies to your own changes as much as to the provider's. If something breaks, point production back at the previous release and fix forward in a new draft. Part 12 on AI agent observability covers the monitoring side.

How Upsolve Approaches Agent Context Versioning

In Upsolve, the unit of release is the workspace. A workspace holds the data model, knowledge base, skills, assets such as golden questions and charts, system prompts and per-user memory, each in its own section. Chat history, a playground and deployment sit in the same sidebar, so context, monitoring, testing and release happen in one place.

The whole workspace is versioned as one release. The data model, knowledge base, skills and assets ship together as a single version, and each workspace keeps production and draft versions. You choose which version to deploy, and deploy it into your product as an iframe.

The demo uses a sample procurement workspace. Its skills include SQL generation, spend details and spend drivers, and its memory section shows that a sample user, Marcus Reid, already has a profile while a sample Japan administrator has not logged in yet. Chat history shows who asked what, from which surface (the playground or an embedded deployment) and at what cost, and any session can be opened by its session ID.

The playground is where a draft gets tested. You can chat as yourself or as any user in your organisation and watch a live trace beside the answer. When the demo asks for monthly PO spend over the last 12 months, the trace shows the agent fetching golden examples, retrieving the schema and validating the chart before it returns the result. As Serguei Balanovich, Upsolve, put it in the demo: "Workspaces allow you to put everything under a single umbrella and deploy everything as a single versioned asset."

Because everything lives in one place, the Builder Agent can read the workspace and its chat history and propose fixes. Those changes land in a draft, so production stays untouched until you promote it. Part 14 on data drift covers that loop, and the Workspaces feature page has the full overview.

Frequently Asked Questions

What is AI agent context versioning?

AI agent context versioning means saving everything that shapes an agent's answers as one immutable, numbered release. That bundle usually includes the data model and metric definitions, knowledge base documents, skills, example questions with trusted SQL, and the system prompt. Each release can be tested before it goes live, deployed to a specific surface and rolled back as a unit. The aim is that any answer can be traced to the exact release that produced it, which turns a vague regression into a comparison between two known versions.

How is agent config management different from prompt versioning?

Prompt versioning tracks the history of individual prompts, often with aliases such as production and staging that point at a specific version. Agent config management covers the whole agent: the prompts plus the data model, knowledge base, skills and examples, and the rules for who edits them, where drafts live and which release each surface runs. Prompt versioning is a useful part of it. On its own it cannot prevent a prompt from referring to a metric that was renamed in the data model the same week.

Should agent memory be versioned with the release?

Usually not in the same way. Per-user memory and chat history accumulate between releases, and rolling them back with the context would discard what users have taught the agent. Keep them beside the release so they can be reviewed, edited or cleared, and record which release was live when each memory was written. Test every new release with at least one user who has rich memory and one who has none, since memory can hide or amplify a context change.

How do you roll back an AI agent?

With whole-context releases, rollback means pointing production at the previous release, which restores a data model, prompts, skills and examples that were tested together. With per-artefact versioning you have to revert each piece separately and check that the old versions still fit, which is slower and error-prone under pressure. In both cases, record the model version in the release, because a provider update can cause a regression that no context rollback will fix.

How do you test a prompt or context change before deploying it?

Make the change in a draft that end users cannot reach. Run your golden questions against the draft and against production and compare both the SQL and the results. Then chat with the draft as several representative users, including restricted ones, while reading the trace for each step: retrieved examples, selected tables, skills used and chart validation. Promote only when every changed answer has an explanation, and monitor live conversations closely after release.

Do I need Git to version agent context?

No. Git works well for files that engineers edit, such as dbt models and skills, and many teams keep agent context in a repository. The requirement is the property Git provides: immutable versions, a clear record of what changed, and a way to restore a known state. Analysts and domain experts often edit descriptions and business rules, so a workspace that versions context without requiring Git can be easier to keep current. What matters is that every piece ships in the same release.

Sources

1. Sclar, Choi, Tsvetkov and Suhr (ICLR 2024): Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. https://arxiv.org/abs/2310.11324

2. Sculley et al., Google (NeurIPS 2015): Hidden Technical Debt in Machine Learning Systems. https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf

3. Chen, Zaharia and Zou: How Is ChatGPT's Behavior Changing over Time? https://arxiv.org/abs/2307.09009

4. Google: Site Reliability Engineering, Chapter 8, Release Engineering. https://sre.google/sre-book/release-engineering/

5. Anthropic Engineering: Effective Context Engineering for AI Agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

6. MLflow Documentation: Manage Prompt Lifecycles with Aliases. https://mlflow.org/docs/latest/genai/prompt-registry/manage-prompt-lifecycles-with-aliases/

7. Sato, Wider and Windheuser, Thoughtworks (martinfowler.com): Continuous Delivery for Machine Learning. https://www.martinfowler.com/articles/cd4ml.html

Try Upsolve

Start free at upsolve.ai with 2,000 free AI credits and build your first versioned workspace, or talk to our team about deploying one agent across your product. Next, Part 05 walks through how to set up a data agent from an empty workspace.