Lately, I’ve been spending quite a bit of time thinking about the concept of an agent “harness.” At first, I was squeamish about the term. It felt like we were creating a new label for something we already had.
This time, though, I think we might be onto something. The concept of an agent harness, the infrastructure layer that wraps, manages, and controls an AI agent’s execution, feels like a clearer mental model to me. I can see how this could become a new area of software engineering.
If an agent is the “brain,” the harness is everything that lets it actually operate in the real world.
Agent → decides what to do next
Harness → makes sure it can actually do it safely and reliably
In practice, and in marketing jargon, we still call this an agent. But we take for granted the software wrapped around an LLM that takes action on our behalf. The LLM’s response is the part that’s already been figured out. Software that parses and makes sense of what the LLM’s response is the new challenge at hand.
You’re already familiar with agent harnesses. Claude Code is an agent harness. Cursor’s IDE is an agent harness. It’s the environment that allows you to prompt an LLM and expect an automated action to be taken based on the LLM’s response. It makes an agent more reliable, safer to use and more scalable.
But the above are either general purpose agents meant to handle almost any task (Claude Code ) or agents more designed to better suite a specific persona like a software engineer (Cursor). Us in data engineering have been waiting for something similar that meets our needs and helps us get our work done.
The Nao Labs Analytics Agent
Nao Labs released an agent earlier this year and made it fully open source. It is geared specifically toward the needs of data professionals. Not only is it an agent, but it also comes with a harness: a UI you can open in your web browser to prompt your LLM of choice (it’s a bring-your-own-keys platform) or connect to external MCP connections or databases.
While the Nao agent harness runs agent loops and orchestrates tools, one thing that is unique about it is it focuses on building a context layer in its folder structure. When you think about Claude Code for example, it requires a lot of guard rails to be put up to properly handle requests more deterministically. This comes in the form of Claude files as well as Skill files. But the main point of those is to give the agent context before it decides on an action to take. Nao takes this part very seriously and instead of over optimizing the agent itself, they suggest to optimize what it knows.
With that, they have outlined a very formal context layer that the agent constantly references. This is more than skill files as it is about offering static assets like schemas, for example. So it’s not just instructions on how to do things, it’s the literal truth of what an agent should use to take action on data.
But what Nao’s agent truly offers is a UI harness you can leverage to do different data tasks. All while using the model of your choice (this would include local models running on your machine as well). It being open source gives you a lot of cost flexibility. You might love the experience of Claude Code, but perhaps you’ve hit limits or you want some additional models choices outside of Anthropic’s. While you can run other models in Claude, an OSS agent is free of roadmap bloat that many a Reddit post has been complaining about with Claude Code.
My Conversation With Claire Gouze
Claire is the co-founder of Nao Labs and someone I’ve been following since Nao came out of the YC 2025 batch last year. While they initially built an IDE (think Cursor for Data Engineering), they also made a very clever parallel agent product. The choice to open-source it impressed me, as did the technical choices they made to improve the agent’s contextual awareness.
I have been talking with Claire about having a conversation for some time, but I also wanted a “boots on the ground” guest who could speak more to AI data engineering and using agents in production. No need to look further than Alejandro Aboy, who knows Nao’s product well and was there to tell us more about what it’s like to build AI in production as a data engineer.
Feel free to watch or listen to the video above, or, if you’d prefer, you can read a condensed version of my questions along with Claire’s answers and Alejandro’s thoughts below.
Note: Below is a condensed version of the above video conversation. Questions and answers are either summarized or lightly edited for clarity and readability.
CORE CONCEPTS
Q: What is an agent and what is a harness?
Claire: I think we started with chat, which was just a conversation with a model that didn’t have any tools and couldn’t really take any action. I think the counterpart to that is the agent, because the agent can take action. So an agent is basically a program based on an LLM, but it also has some tools and can take action. That action could be doing some research and answering the question, or answering the question and then taking actions in your own applications, your database, or anything else.
I guess the harness is everything. It’s kind of like the box where you put your model. So I would say an agent is, in some form, a harness on top of your LLM. Then you can add a lot of components to your harness, like the context, the MCP you’re going to give it, and the skills you’re going to create. All of these components sit on top of the LLM and also define the reliability and the output of the agent.
Alejandro: Three years ago, I had an idea that changed a year ago. And a couple of months ago, it kept changing. These days, basically, I would summarize it as a system you put in place, driven by an LLM, and you can give it the world to figure out what to do. And that world is the harness.
When I say that two years ago or one year ago, I was thinking of a different idea, because we didn’t have the same things, like memory tools and MCPs. If we go in the analytics direction, the way progressive disclosure works with semantic layers and discovery models, we’re in that reality. It’s something that changed a lot. I would say mostly in the last six months, it grew a lot.
Q: Are there separate “agent engineers” and “harness engineers”?
Claire: I think agents are really fundamental. I think not many people recreate agentic loops. For example, people building agents today mostly use frameworks like LangChain, which always provide you with the agentic loop. And so I see teams working more on the harness where they put their agentic loop. For example, they have LangChain for the agentic loop and then, on top, they’re going to build a repository with context, build a UI, and plug in skills and MCP. So that’s where they put most of the effort.
NAO LABS’ STRATEGY
Q: What was Nao Labs’ original premise and why create an open source agent?
Claire: Before building this <agent>, we were working on an IDE specialized for data analysis. And what we realized is that what data teams were mostly excited about was more like analytics agents for the whole company, rather than just improving their own agents for development. So that was the main issue we were seeing: they wanted to do analytics agents for the whole company. But there were also a lot of solutions doing that already. Every BI tool was going to do this. Every warehouse is doing this, like Cortex, Databricks’ Genie. They’re all onto this issue, and there are a lot of startups already working on that. So we were thinking, that’s very weird because there are so many vendors trying to do that. But when you talk to data teams, not many have already started to implement it. That was back around November.
And so we were wondering why, and we realized that all the solutions in the market were closed-source: a bit of a black box, often making you use their own ontology or their own semantic layer. And so it required a lot of effort, often a lot of money. In the end, if it was not working well, you couldn’t really tweak it to make it work for you.
So we thought there’s an opportunity here to create a product that is way more transparent in the way it’s built, and also doesn’t cost you to invest in it. So that’s why we created a framework that is open source. It’s free to have an MVP on it, and it’s also free to customize and to make your own. This way, you can debug it until it’s really reliable and really worth investing more time and money into.
PRODUCTION RELIABILITY & MAINTENANCE
Q: How reliable are open source agents in production, and what’s the maintenance overhead?
Claire: I think we should not lie about the fact that there is some work to do to have a reliable agent, but I think that’s the truth for any agent. And I think other closed-source vendors can falsely make you believe that you just need to plug in your data and then it’s going to be reliable because they have the secret sauce. But the real secret sauce is the work that you, as a data team, are going to put into it, like all the data modeling you’re going to do, and all the data documentation you’re going to do. That’s actually the real value. So you’re the one bringing the value into your own agents.
But our philosophy is that you should not need to do full prep work before actually having an agent running. So you can just plug in what you have, whatever data you have, whatever data documentation you have. And we have an evaluation framework that will tell you, with what you have currently, that you have 5% reliability, which is bad. And we're going to show you where the agent fails. And then you start the work of improving your data and modeling your data documentation. Maybe, if you want, you can add a semantic layer. And at some point, you get to good reliability. I would say, for me, for example, I have 15 tables in my core layer. And maybe it took me four or five days to make it from 5% reliability to 85% reliability. So it takes some time, but it's the work you need to do.
Alejandro: It’s funny because my technical profile changed dramatically at a point where, depending on the week, I am a data AI engineer. And depending on the other week, I am an AI data engineer. Some weeks I’m not even a data engineer.
<Claire> highlighted something quite important. I spend 70% of the time that I’m working on my agent projects at work. 70% of that time I spend on the observability tool, coming up with skills to work on top of the observability, scoping new evals, deprecating the old ones, calibrating for saturation.
Most of the time I spend there is on the observability tool. And the main thing is that if you get what I call human readiness (instead of AI readiness) clear before jumping into any of it, observability will make sense, and then all the decisions will make sense. And the remaining 30% or 20% is when you touch the prompt, when you touch the skills of the agent, the memory, whatever other thing, the harness. And that’s what I’ve been living over the last year, mostly.
If all the things were already a mess when it comes to data (specifically all the descriptions, semantics, everything) before AI, it will be even more of a mess these days. And it’s really hard for companies not to fall for the shiny object these days, rush into putting the agent together, get awful answers, and say this AI is not working.
So yeah, that’s my main experience that I’m living. And I have a customer-facing main agent project that is running queries, calling the backend, and doing a mix of stuff. It’s not using the Nao part yet (not in production); I’ve been testing a lot outside of it. But yeah, the main blocker most of the time is: how do I ensure something that is not deterministic is as deterministic as possible, so I don’t get customers jumping down my throat when they ask the same question and get three different answers.
WHERE AI SHOULD BE USED
Q: How encompassing should AI be? Should it write all pipelines, or just at checkpoints?
Claire: Yeah, it’s funny because there was both Anthropic and OpenAI wrote an article about how they built their in-house agents and they both came to the same conclusion, which was the main part that humans still should be writing are metrics definition. And I think that’s right. That’s the only middle checkpoint that humans should really do by hand almost. I think it can be accelerated by AI. So if you think you of analytics engineers, they can use like code code, cursor, whatever, to write a data model. But when it comes to defining the metrics, then they really should take some time to think about it, like think with all the business stakeholders and write those definitions themselves.
And then once this is done, I think the agent can like take this definition and actually use them in the analytics agents.
Hoyt: Yeah, the other thing that Anthropic basically admitted was if the data model sucked, then this sucks. So it really was pretty straightforward there, wasn’t it? It was like nothing has actually changed.
Claire: Exactly, and I had the same conclusion. I think I did a lot of work on the context of my agent. And at some point I was at a 50% reliability glass ceiling because my data model was ambiguous. There were some fields missing, several sorts of truths for one metric. And only when I reworked my data model, then I reached the 80 plus reliability. So there’s no secret sauce. You still need to do data modeling.
CURRENT STATE OF ADOPTION
Q: What are early data team adopters excited about with AI?
Claire: I think they’re mostly excited because of what you just said. People don’t talk to them anymore. They just talk to the agent.
Because I think Data Teams had already accelerated their own work using AI agents. It was working well for them because they can challenge the work of the agent, add some rules so that it’s getting better and better. But they need to do this work for other people so that if they just write the most obscure sentence, the harness will make it that it understands what the user was meaning.
I think what they really love is that they literally see the number of analytics requests cut by, I don’t know, 80% in some cases. I think some of the biggest adopters of Nao are actually company CEOs. A lot of them just chat with Nao’s agent all the time to get all the metrics. So that’s pretty fun to see, honestly.
Q: Are data teams building context structures around their data?
Claire: Yeah, well first they look at the CEO conversations, because they want to make sure the agent is not mistaken. So, I mean, for now, I think they’re still at the phase where they look a lot at the agent answers, and have these feedback loops of, okay, maybe the agent did give a wrong answer, or the CEO had to actually course correct the agent, so that’s not ideal.
And so there’s this whole feedback loop on the context and agent quality. I think that’s one thing, and my hope is that it’s going to converge, and at some point, the agent knows most things, and you don’t need to do this work anymore.
But what I see them doing is they do a lot of complex analysis scenarios of, I don’t know, pricing elasticity of like forecasting, this kind of thing, which are way more complex than what they used to do before, because they didn’t have time to do that.
SKILLS & CONTEXT IN PRODUCTION
Q: How do skills help or hurt in production?
Alejandro: Depending on how non-deterministic the skills are. But I, for example, I’m working on implementing Metabase MCP for non-technical users, because they wanted to have it in Claude.
The first concern was, okay, if I just enable it and let them ask anything, there’s no guidance. How to semantically search for stuff, how to pick up the right thing, how to do the progressive disclosure. So I build a skill, but then I have a meta skill that reviews skills based on best practices, papers, and other things.
Then I keep coming up with scenarios and tests to run a bunch of times, and get these avals going that we can define later. The main thing in production, when the thing is actually working, is that if you don’t have a system in place to review those chats, as Claire was saying, it’s kind of really hard to get any notes or conclusions. Okay, this is drifting, it’s getting what it wants. I am packing too much on one skill. I have one skill, but then I have 10 references inside that skill, so it has a lot of things to check.
And you start asking yourself a lot of questions on how you are packaging the use case and the skill. And most of the time, the answer is you’re expecting way too much from the skills, because they’re really good. They are wrapped prompts at the end of the day, but the same thing with a prompt. If you ask more than one thing, it can hold two, maybe three. But if you ask a lot of if-else, a lot of research is conditionals, that’s where the skills are not really a silver bullet in production mostly.
Q: What is “context” in production?
Alejandro: The way I see it, you have kind of the static and dynamic shape of context - this is how I like to perceive it. When we talk about things like the harness, they are all being injected into an agent’s context. All the tools you can use, everything you can plug in when you’re starting up Claude Code-all those things are going into the context of Claude Code.
So there you have kind of the built-in context, and it’s static because it will always look the same during the sessions. And then whatever is dynamic: when you are running queries, you are discovering models, then running a query on top of the model. You’re reading the description of a model to know if the use case you want to cover can be covered by that model, and then running all the tools and working more sequentially on top of those outputs. So whatever you can give the agent in a static or dynamic way to know what the best and right action to take next is, that’s context, by my definition.
EVALUATION & TESTING
Q: What does an eval look like for agentic data work?
Claire: I think there are a lot of different paths you can take. I would say the two main ones are deterministic evaluation and LLM-as-a-judge, which is a bit more complete but also a bit less trustworthy. We decided to go with deterministic evaluation because we really want to make sure the evaluation does not hallucinate. So basically, we have you write unit tests with questions and the expected SQL.
Then we execute that SQL and get the answer from it. On the other side, we send the question to the agent and tell it to fill in a table with the same schema. Then we do a data diff of the two answers, so we don’t really care about the SQL that was written—we care about having the right number.
So that’s a way for us to really measure whether the number is right for us. We don’t really care about pass/fail. And I think another way to do eval is LLM-as-a-judge, so the LLM can tell you, like, “Okay, does it seem right? Did it actually use the definition we wanted?” But you can’t really be sure what’s going on with LLM-as-a-judge. So I think that’s not the way to go, or it’s complementary, but not the core eval.
THE “COMPANY BRAIN” EXPERIMENT
Q: What is your Company Brain project?
Claire: Yeah, so at first it just started as an experiment because everyone was talking about a company brain, but I didn’t see anyone really implementing it or making it work. So I felt like I’m going to try to make it work. And I was already doing a lot of linking and writing with the help of Claude. I had good results by scraping all my LinkedIn posts and creating a skill from that. So I said, I’m going to do all of that, but version it in GitHub so that every time I work, I learn from what I do.
So I just created a full GitHub repo with one folder per domain. Now, every time I do something, I do it in Claude; Claude logs everything we do. It has skills for each domain, and every time there’s a feedback loop to improve the skill. So I think it’s working pretty well. I’ve gained a lot of time: I don’t write notes anymore. I don’t do my to-dos anymore. That’s all done, or I create issues on GitHub. That’s just all done. And I can focus on prioritizing, syncing, and the important human-context things I still need to do.
TOKEN ECONOMICS & FUTURE
Q: How fast is AI improving and how do you keep up?
Alejandro: I split this in two different worlds. The worlds where people like us live, where we are on the latest Anthropic announcement, or the last new AI harness, password and keyword, al those fancy keywords that we see every day. And then the other world, which is the enterprise and companies, not startups, but more like old school companies. Which is they are using AI like it is 2022 or 2023 still. So every time I make a decision on what to implement, what to go for, what to pursue, I try to close that gap and I end up realizing that, okay, I cannot go in this direction because it’s too far away from someone who is barely understanding how to prompt an agent.
And I’m talking about customer-facing. I’m also talking about multi-tenant situations where you need to handle an agent that covers routing and authentication layers and more complex stuff.
Claire—this is something I’m interested in—is that, next to Anthropic emails for announcements, I see Nao emails most of the time because you’re achieving something new every week with things like MCP apps or exposing Nao Agent as MCP.
Claire: I would say that every week we try to ship something that we think is innovative and that we want to put in the product. For example, the MCP app is something that nobody really requested, but we decided we needed to go in the direction where your data is available everywhere and the UI becomes mainly your agent. That was an innovation that we really pushed for, and I think we were the first to actually do that for our BI tool.
And then we also receive a lot of issues and feature requests from our users, which I would say are less innovative. It’s more like recreating everything you need for a BI tool, like connectors, charts, and filters. That’s less exciting, but we still need those to make the product work. And since we’re open source, what’s great is that some people just code them for us. So that’s cool. But we try to bring some innovation every week because that’s what people are looking for.
Q: How much coding are you actually doing, and what if token costs 10x?
Alejandro: I’m only deleting stuff that I can get Claude Code to delete when I tell it to. That’s the closest to coding. We’re like, “Please remove that.” Okay, we’ll derail the conversation and go to something else. Well, that’s the only moment that I’m touching code.
Yeah, we are over-relying, I would say. And that’s given a lot of speed and productivity bumps. It’s also giving us the sensation that the day never ends because you can always keep jumping into something else, which is kind of messing with our brains, I would say. But if they give a big bump in prices, let’s say 10x tomorrow, I would say we either go in the open-source direction with Mac minis or equivalent good computers to install models. I’ve been experimenting with the Qwen models, Kimi 2.5. The benchmarks always look really good, but you know that you can overfit on benchmarks. So that’s another conversation.
So yeah, I think open source, once again, as at any other point in time, could be a direction to pursue, and also a chance to realize how much you really make a difference when code is out of the question, and then, when code is back, whether you’re still relevant or not. Because I think we’ll need to kind of remove a lot of dust from our brains if that happens.
Claire: Yeah, I mean, I think the tricky thing is that people are not yet in the phase of optimizing their work. I think they are in the phase of, like, trying to prove that the agent can actually do the work. For example, with my company brain, I don’t optimize for the number of tokens. I just want it to work. That’s what I’m trying to achieve. And on some topics, it doesn’t even work yet. Or it only works if I give it a lot, a lot of context. So if tomorrow I need to reduce the context, then my agent is not going to work anymore. Well, then it breaks the purpose of having the agent.
So I guess if tomorrow the price of tokens is 10X, I will be just using way less just because then it would not work anymore if I just optimized for the cost. So I think that would be counterproductive in some ways.
But I think at some point, when we are all reaching a good level of quality and reliability of the agents, then we need to go on to this optimization phase, but I don’t think we are there yet.
STARTING WITH NAO’S OPEN SOURCE AGENT
Q: How do you successfully use an open source agent to start?
Claire: Yeah, I mean, I think the advice I usually give is really to start small. I’ve seen some data teams that just plug the agent into the whole warehouse, and then they don’t know what to test. They can’t test everything, all the metrics from all the tables. So my advice is to pick at most 20 tables, know the key 20 questions you want the agent to get right on these tables, and then make sure it’s 100% (or close) reliable for those.
And I think what I found to be the biggest jumps in reliability were improving data modeling and also improving the context around what data analysts would do. Like, with this question, I know it goes to this table because this table is the source of truth for this kind of domain or metric. So really trying to orchestrate the agent the same way data analysts would helped a lot. And what I did to help other teams is we created some skills that will help you build your context according to all the findings I’ve made.
So I think it now helps teams start at about 50% reliability instead of about 5%. Then the race is like, “You have to rework your data model. I can’t do that for you.”
Links:
Hi, my name is Hoyt. I’ve spent different lives in Marketing, Data Science and Data Product Management. Other than this Substack, I am the founder of Early Signal. I help data tech startups build authentic connections with technical audiences through bespoke technical content and intentional distribution. Are you an early stage start up or solopreneur wanting to get creative with your technical content and distribution strategy? Let’s talk!
