The Complete Agent Stack · Part 06 of 14 · Agent harness layer

This guide is one part of a 14-part series on what a production data agent needs, from the data model it reads to the agent that keeps it accurate. Each part stands on its own, and they build on each other in order.

← Previous: Part 05: How to Set Up a Data Agent · Next: Part 07: MCP Servers for Data Agents →

Two teams can run the same model against the same warehouse and get very different results. The difference usually sits in the harness: the planning loop, the tools, the instructions that load for a task, the checks that run before an answer is shown and the memory that carries over. This guide explains what an AI agent harness is, why hardcoded workflows struggle with analytical questions, and how skills and per-user memory change what a data agent can do.

Key Takeaways

• The harness is everything except the model: LangChain defines an agent harness as "every piece of code, configuration, and execution logic that isn't the model itself", and reports moving its coding agent from the top 30 to the top 5 on Terminal Bench 2.0 by changing only the harness.

• Agents need ground truth at every step: Anthropic's guidance says an agent should gain "ground truth" from its environment at each step, such as tool call results or code execution, which for a data agent means running and checking SQL before it answers.

• Self-correction needs an external signal: An ICLR 2024 study found that language models struggle to correct their own reasoning without external feedback and sometimes get worse, so a data agent's recovery works best when database errors and result checks drive it.

• Skills load only when relevant: Anthropic's Agent Skills load a short name and description at startup and the full instructions only when a task calls for them, so one harness can hold many specialised analyses without filling its context window.

The 45-second version:

What Is an AI Agent Harness?

An AI agent harness is the runtime that turns a language model into an agent. The harness gives the model a system prompt, tools, a loop that executes the tool calls it requests, a way to manage context as the conversation grows and somewhere to keep state between sessions. LangChain summarises the relationship as "Agent = Model + Harness", and lists system prompts, tools, skills and MCP servers, bundled infrastructure such as a sandbox, orchestration logic, and hooks for deterministic steps such as compaction [4].

Anthropic uses the same term for its own tooling. Its write-up on long-running agents describes the Claude Agent SDK as "a powerful, general-purpose agent harness" and is mostly about harness design: a feature list, a progress file, git commits for state and end-to-end testing. "Each new session begins with no memory of what came before," so the harness has to carry state that the model cannot [2].

A data agent's harness takes a question from someone who does not know the schema, finds the right tables, writes SQL, checks the result, picks a chart and explains the answer in business language. Each of those steps is a harness decision. In The Complete Agent Stack, the harness sits above the ontology (the data model and row-level security) and below the surfaces where people meet the agent. The data model gives the agent correct definitions, and the harness determines whether the agent uses them well.

Why Do Hardcoded Agent Workflows Break?

Many early analytics assistants were fixed pipelines: classify the question, retrieve tables, generate SQL, run it, chart it, summarise. Anthropic calls systems in which LLMs and tools follow predefined code paths "workflows", and systems in which the model dynamically directs its own process and tool use "agents" [1]. Workflows offer predictability for well-defined tasks. Agents suit problems where the number and order of steps cannot be known in advance.

Analytical questions from non-technical users belong in the second group. "Monthly spend trend by category" might need one query on a clean table, or several steps on a table that stores dates as text. A fixed pipeline runs the same sequence either way, and when the data breaks its assumptions there is no step for recovery.

A tool-calling harness hands control to the model, which plans, calls tools such as "run SQL" or "render chart", reads each result and decides what to do next. Anthropic notes that agents often trade latency and cost for better task performance [1]. That trade depends on the workflow being replaced, because a planning loop can skip stages a simple question does not need. When Upsolve replaced its pre-hardcoded workflows with tool calls and skills, the rebuilt harness became twice as fast.

A team with three fixed reports may be better served by a workflow. A team that wants anyone to ask anything needs an agent, plus the observability to see what it did.

What Goes Into a Data Agent Harness?

Four components carry most of the weight in analytical work, whichever model sits in the middle.

Tools and a planning loop

Tools are the agent's verbs: list tables, read column descriptions, fetch a column's distinct values, run a query, render a chart. Anthropic treats the "agent-computer interface" as a design surface and recommends thorough tool documentation and testing [1]. A good data tool returns something the model can act on, such as a schema with descriptions or a result set with its row count.

Validation and self-correction

A query that runs can still give the wrong answer, and a query that fails can usually be fixed. SQL provides unusually rich ground truth: the database returns syntax errors, type mismatches, empty result sets and row counts, and the harness can add its own checks before anything is charted.

Research explains why this signal matters. Huang and colleagues found that LLMs struggle to self-correct reasoning without external feedback, and sometimes get worse after trying [6]. Text-to-SQL systems that recover well use execution feedback instead. MAC-SQL, presented at COLING 2025, pairs a main SQL-writing agent with a Refiner that uses external tools to repair erroneous queries, and the full framework reached 59.59% execution accuracy on the BIRD test set with GPT-4 [7]. The practical lesson is to give the model a real signal and a bounded number of retries, then explain clearly when it still cannot answer. Our guide to self-improving agents covers how to bound that recovery loop.

Skills

A skill is a packaged set of instructions, sometimes with scripts or reference files, that an agent loads when a task calls for it. Anthropic defines Agent Skills as "organized folders of instructions, scripts, and resources that agents can discover and load dynamically" [3]. They work through progressive disclosure: the name and description load at startup, the full instructions load when the skill becomes relevant, and bundled files load only as needed. Anthropic has also published Agent Skills as an open standard for portability across platforms [3].

Memory

Memory is the state that survives the end of a conversation. The CoALA framework for language agents separates working memory, which holds information for the current decision cycle, from long-term episodic, semantic and procedural memory [5]. In a data agent, the data model plays the role of semantic memory and skills behave like procedural memory written as text. Our guide to AI agent memory covers memory architecture in depth.

Agent Skills: Encoding the Analyses Your Business Runs

A general-purpose agent knows SQL and charting. It does not know that a finance team explains every spend change the same way, or that a churn question always means a cohort comparison. Skills hold that knowledge, and they are how a builder makes a generic agent specific to one business.

Skills that work in production tend to share four properties:

• A precise trigger. The description uses the words users actually type, because it is the only part the model sees before deciding.

• A method the agent can adapt. The body describes the steps of the analysis and leaves the agent to write SQL against the current schema.

• A defined output shape. The skill says what comes first, usually the headline number, and which caveats belong in the answer.

• A narrow scope. One analysis per skill keeps each one testable and easy to retire.

Take a skill that a finance team might write to explain spend changes. It tells the agent to frame the comparison, establish the headline figure and then decompose the change to find where it concentrates. Asked why spend fell from one month to the next, an agent with that skill reports the headline change first and then breaks it down by category, so every answer to that kind of question follows the team's method.

The harder question is where new skills come from. Anthropic looks ahead to agents that "create, edit, and evaluate Skills on their own" [3]. In analytics, a review step is worth keeping, because an agent that rewrites its own method without oversight can drift towards whatever its most recent users wanted. A practical middle path is for the agent to propose skills from real conversations and for a builder to approve each one. That keeps a person accountable for the method and connects the harness to the improvement loop covered in our guides to golden questions and data drift.

Agent Memory: Personalising Output Without Changing the Numbers

Per-user memory raises a fair concern in analytics: if an agent remembers things about each user, two people could get different numbers for the same question. Whether that happens depends on what memory is allowed to touch.

A useful rule is to separate presentation from substance. Chart type, colour, default time bucketing and the level of detail in explanations can safely vary by user. Metric definitions, filters, permissions and the SQL logic behind a metric should come from the shared data model and security layer, never from one person's memory. Two users asking the same question then see the same figures, presented the way each prefers.

Scope and hygiene matter as much as content. Anthropic's memory tool documentation has the application map memory onto storage it controls, "such as a per-user directory or keys in a database", and recommends stripping sensitive information, capping file sizes, expiring memory that has not been accessed in a long time and validating every path [8]. The same principles apply to any data agent: scope memory to the user, keep it inspectable and let it expire.

How Upsolve Approaches the Agent Harness

Upsolve rebuilt its agent harness from scratch for people who do not know the underlying data: internal teams asking ad hoc questions and, when the agent is embedded, a product's own end users. Harness 2.0 picks the right tool or skill for each step of a question and does the work in the background: it plans, writes SQL, validates the query and the chart, and corrects itself, so what reaches the user is analytics the agent has already checked as clean and usable.

In the procurement demo project, a sample user, Grace Tan, asked for monthly spend trends by category. The trace in the observability view records each step: fetching skills, generating SQL, retrieving spend details, running SQL validation and chart visualisation, then rendering a bar chart. Along the way the agent misread how the dates were stored, so it removed its time bucketing, treated the dates as strings and recovered without the user doing anything. The trace view is the subject of our guide to AI agent observability.

Generate SQL and generate chart skills ship by default and improve over time, and builders add custom skills for their own analyses, such as the spend-change skill described above. As people use the agent, it proposes new skills drawn from real conversations. Suggestions never change the agent automatically. They wait for a builder to review them and add them.

Per-user memory records preferences such as chart types, colours and time bucketing. In the demo, another sample user, Carlos Ruiz, had saved preferences for bar charts, orange as the chart colour and monthly bucketing, and the same question produced a chart in that style with the same analysis underneath. "It still delivers the same analytical accuracy, but we take the user preferences into account when we produce these results," said Serguei Balanovich, Upsolve.

The harness runs wherever the agent is deployed: embedded in a product, in Slack, in Claude through MCP (see data analysis in Claude with MCP) or inside a team's own agent harness. It can also call external tools through MCP servers. The full feature list is on the Harness 2.0 feature page.

Frequently Asked Questions

What is an AI agent harness?

An AI agent harness is all the software around a language model that makes it behave as an agent: the system prompt, the tools and their descriptions, the loop that executes tool calls, context management, skills, memory and any deterministic checks. The model provides reasoning and language. The harness controls what the model sees, what it can do and when a task is complete. For data agents, it also decides how SQL is validated and how results become charts.

Does the model or the harness matter more for accuracy?

Both matter, and the harness is often the cheaper lever. LangChain reports moving its coding agent from the top 30 to the top 5 on Terminal Bench 2.0 by changing only the harness. For data agents, harness choices such as validating SQL against the database, loading the right skill and grounding queries in a governed data model address the failures users see most. A stronger model cannot supply business context or execution feedback that the harness never provides.

What are agent skills?

Agent skills are packaged instructions, sometimes with scripts or reference files, that an agent loads when a task needs them. Anthropic's Agent Skills use progressive disclosure: only a short name and description sit in context by default, and the full instructions load when the skill is relevant. In a data agent, a skill usually encodes one analysis the business runs often, such as explaining a change in spend, so every user gets the same method.

How does an AI agent correct its own mistakes?

Reliable self-correction depends on an external signal. Research presented at ICLR 2024 found that language models struggle to fix their own reasoning when asked to reconsider without new information. Data agents have a strong signal available, because the database returns errors, empty results and row counts. A well-built harness feeds those back to the model, allows a bounded number of retries and explains clearly when it still cannot give a trustworthy answer.

Does per-user memory make AI analytics answers inconsistent?

It should not, provided memory is limited to presentation. Chart type, colour, time bucketing and level of detail can vary by user without affecting the figures. Metric definitions, filters and permissions should come from the shared data model and security rules, which memory never overrides. Memory should also be scoped to each user, inspectable and set to expire, in line with the safeguards Anthropic recommends for its memory tool.

Should an agent be allowed to write its own skills?

Agents are good at noticing patterns in real conversations, so they are a useful source of candidate skills. Letting them change their own instructions without review is riskier, because a flawed method spreads to every later answer. A safer pattern is for the agent to propose skills and for a builder to approve, edit or reject each one, so a person stays accountable for what the agent treats as standard practice.

Sources

1. Anthropic: Building Effective AI Agents. https://www.anthropic.com/engineering/building-effective-agents

2. Anthropic: Effective harnesses for long-running agents. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

3. Anthropic: Equipping agents for the real world with Agent Skills. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills

4. LangChain: The Anatomy of an Agent Harness. https://www.langchain.com/blog/the-anatomy-of-an-agent-harness

5. arXiv / Sumers, Yao, Narasimhan and Griffiths: Cognitive Architectures for Language Agents. https://arxiv.org/abs/2309.02427

6. ICLR 2024 / Huang et al.: Large Language Models Cannot Self-Correct Reasoning Yet. https://arxiv.org/abs/2310.01798

7. COLING 2025 / Wang et al.: MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. https://arxiv.org/abs/2312.11242

8. Claude Docs: Memory tool. https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool

Try a harness built for people who don't know the schema. Start free at upsolve.ai with 2,000 free AI credits, or talk to our team about deploying a data agent in your product, Slack or Claude. Next in the series: Part 07: MCP Servers for Data Agents.