On Harnesses and loop-engineering in AI

Introduction

The past years discourse in AI has been rife conversations about harnesses. Everyone seems to be talking about a Harness, the importance of it, and how the Harness may in fact end up becoming even more important than the LLM itself. But, what really is a Harness? Here, we deep dive and understand what a harness is, and how it is constructed, by using some popular ones as case studies.

Perhaps, the easiest way to view a Harness is that it is all the code in an agent surrounding the core LLM.

More formally, a harness is the engineered system around a language model that allows it to interact with other tools and APIs. The model decides which tool to call. The Harness dictates how this tool call is executed. That is, the harness decides what the model can see, what it can do, what it remembers, when it is allowed to act, and how its actions are turned into changes in the world.

This distinction matters because the same model can behave like very different products depending on the harness around it. Put a frontier model behind a chat UI, and you get ChatGPT which answers questions. Put it behind a filesystem, shell, editor, patch tool, test runner, and permission system and it becomes a coding assistant like Codex. Put it behind a browser controller and screenshot loop and it becomes a computer-use agent. Put it inside a cloud VM with persistent state, task tracking, and recovery and it starts to look like a remote worker.

In previous posts, we looked at two pieces of Claude Code's tool in detail: the tool architecture and the memory architecture. This post zooms out. The goal is to define the general shape of a harness and build a taxonomy that applies not just to Claude Code, but also to Codex-style coding systems, OpenManus-style remote workers, browser-use/OpenClaw-style computer-use agents, and Hermes-style general agent runtimes.

A Minimal Harness

At the center of every harness is a loop. The exact implementation may vary, but the structure is often the same: observe the environment, construct a prompt, call the model, parse its response, execute an action, return the result, and continue until the task is done or the system decides to stop.

This loop is simple to grasp, but the devil is in the details of each box: What counts as an observation? Which files are included? How are tool schemas described? How are screenshots translated into model-readable state? What happens when the model calls the wrong tool? Who approves dangerous commands? How is the conversation compacted when the context window fills up? How does the system recover after a failed edit?

Answers to those questions are architecture design of the harness. The LLM is important, but the harness defines the world the LLM inhabits.

The Components of a Harness

A useful way to understand modern harnesses is to split them into five layers:

Most harnesses are composed of these components, with varied emphasis on each compoenent. For example, a local coding harness emphasizes files, shell commands, patches, tests, and permissions. On the other hand, a browser harness emphasizes screenshots, DOM state, clicking, typing, and page navigation. Similarly, a remote worker emphasizes persistence, task tracking, and recovery, and a personal agent runtime emphasizes memory, integrations, scheduling, and extensibility.

Overall, these components are trying to answer the questions: what can it do, what does it remember, what does it see, how does the loop run, and how is the system exposed to users?

Layer 1: Tools / Action Layer

The tool layer is the most visible part of a harness. It defines the actions available to the model. In a coding harness, these actions might be reading files, editing files, running shell commands, searching the repository, applying patches, running tests, or interacting with git. In a browser harness, they might be clicking, typing, scrolling, navigating, taking screenshots, inspecting DOM nodes, or extracting page content. In a general agent runtime, tools might be API calls, calendar actions, email actions, database queries, web searches, or custom functions registered by the developer.

A mature tool layer would include not only functions, but also schemas, validation, permissions, result mapping, concurrency rules, and UI rendering. The harness must decide which tools are available, how they are described to the model, whether a proposed call is valid, whether the user needs to approve it, how the result should be summarized, and whether multiple calls can run in parallel. This is why "tool use" is a necessary but incomplete description of agents.

Layer 2: Memory / Persistence

The memory layer tries to persist the experiences of the harness over time: what should be learned from the current model/tool call? Can it be used in future tool calls?

There are several kinds of persistence. Some are explicit memories: user preferences, project facts, long-term notes, background summaries, and learned conventions. Some are session artifacts: transcripts, task logs, resumable conversations, file snapshots, browser state, or VM state. Some are operational: cached titles, checkpoints, approval decisions, tool history, and recovery metadata.

Memory is related to context construction, but it is not the same thing. Memory is stored information. Context construction is the act of deciding what stored information, current environment state, and instructions are shown to the model on this turn. A memory can exist without being included in the prompt. Conversely, most context is not memory: a git diff, a selected code region, a screenshot, a tool schema, or a slash-command expansion may be included right now without being stored long-term.

This separation is important. Models do not operate over the world directly. They operate over a constructed representation of the world. Memory is one source for that representation, not the representation itself.

Layer 3: Context Construction

Context construction is the least visible layer and often the most opinionanted one. Since Large Language Models do not have strong memory capabilities, the context construction must serve the all important job of deciding what does the model see before it acts?

In a coding harness, context construction may include the system prompt, model-specific instructions, tool descriptions, project instructions, current working directory, git status, open files, selected editor text, recent terminal output, repository search results, user attachments, and memory snippets. In a browser harness, it may include a screenshot, accessibility tree, DOM summary, current URL, visible text, previous browser actions, and task objective. In a remote worker harness, it may include the task description, VM state, plan, task history, artifacts, browser state, and previous failures.

Layer 4: Agent Runtime / Control Loop

Agents, like programs have a control flow. That is, a tool is called, things are passed to the model to make a call. The result of the model is to call another tool, and so forth. In a way, this is very similar to a program where control elements like if...then...else dictate what function will be called next. The runtime layer's role is to enable these.

A basic runtime can be as easy as a while loop around a model call and a tool dispatcher. However, a runtime may have more responsibilities. These may include handling streaming, retries, model fallback, max-turn limits, rate limits, token budgets, cancellation, interruption, tool-result insertion, background tasks, subagents, and among others.

Compaction belongs here rather than in memory because it is not primarily about long-term recall. Large language models, unfortunately, are not capable of long-term recall. Instead, the runtime must decide when the current conversation is too large, what can be summarized, what must be preserved verbatim, and how to avoid losing task-critical state. These are opinionanted design decisions in code.

Bottomline, the runtime differentiates a single model response and an agentic run.

Layer 5: Product Surface / Extensibility

This surface takes all the above machinery, and builds a product around it that people can use. It includes the terminal UI, web UI, editor integration, session list, task view, permission prompts, progress indicators, notifications, logs, remote control, plugin system, skill system, configuration files, and account integrations.

Think of this layer as the UI of the agent. The product surface defines trust boundaries. It is where users approve dangerous actions, inspect what happened, resume old sessions, recover from mistakes, install extensions, configure MCP servers, and understand what the agent is doing. For coding agents, this layer decides whether the assistant feels like a black box or a controlled collaborator.

Extensibility also belongs here. Skills, plugins, MCP servers, and custom tool registries are the way harnesses grow new capabilities without changing the core model.

Deep Dive

With the components in place, we can now look at how real harnesses make different tradeoffs. The same five layers show up across systems, but each family emphasizes a different environment, control loop, and product surface.

The following examples are not meant to be exhaustive. They are meant to make the abstraction concrete: local coding agents, remote workers, browser agents, and general agent runtimes all wrap models in different kinds of software bodies.

Four Harness Families

The five-layer taxonomy is general, but different harness families emphasize different layers. Here are the four categories worth comparing.

Claude Code / Codex

Claude Code and Codex-style systems are local work harnesses. Their main design choice is to treat the developer's machine as the agent's environment. Let us go over the different components of these harnesses one by one.

The tools for such a harness are designed around a repository, a terminal, a filesystem, and often an editor. It can inspect files, edit them, run commands, search the codebase, execute tests, and summarize the resulting diff.

Such harnesses operates on a live workspace where mistakes can corrupt files, delete work, or run dangerous commands. That is why they invest heavily in permission checks, read/write distinctions, shell gating, patch application, command rendering, and user-visible diffs. They have to expose enough power for useful work while keeping the user in control.

The context construction layer for a coding model needs details like project instructions, current directory, git state, visible editor context, recent terminal output, tool descriptions, repository search results, and sometimes memories or saved project conventions. The harness chooses which of these become part of the prompt.

The runtime in this operates on coding trajectories. The loop looks like: inspect files, reason about the change, edit, run tests, observe failures, edit again, and stop when the result is good enough. The hard runtime problems are interruption, retries, context compaction, max-turn limits, partial failures, and transcript persistence.

The product surface for a coding agent includes terminal rendering, progress indicators, todo lists, permission prompts, session resume, plugin loading, MCP servers, and remote bridges are all part of the harness. They determine whether the system feels like a transparent collaborator or a black box that occasionally mutates your repository. If you've worked with Claude Code and Codex both, you can tell that one certainly feels much more like a collaborator than the other.

OpenManus

OpenManus is a good example of the remote-worker style of harness, even though it should not be treated as proven architectural evidence for proprietary Manus. At a zoomed-out level, OpenManus includes a general-purpose agent wrapped around tools for Python execution, browser use, file editing, human asking, termination, and MCP-provided extensions.

The tool layer here is broader than a coding harness but less specialized. OpenManus exposes a Python executor, browser tool, string-replace editor, human-question tool, terminate tool, and dynamically connected MCP tools. This makes it a general task worker rather than a pure coding system. It can compute, browse, edit workspace files, ask for missing information, and expand its capabilities through external servers.

The planning flow is of utmost importance for a remote-worker. And so, OpenManus has a planning layer that creates an explicit plan, marks steps as in-progress or completed, chooses an executor for the current step, and runs that executor against a step-specific prompt. This is a different emphasis from Claude Code, though both have plan modes, they are both designed to create very different plans. A local coding harness often centers around the immediate user request and repository loop; a remote worker harness centers around task decomposition and durable progress.

Memory and persistence are present, but the focus for persistence is on ensuring the harness carries task state through steps. To achieve this, it advances through a managed sequence: plan, execute current step, mark progress, continue. That is why remote-worker harnesses need recovery and supervision surfaces. If the task is long-running, the user needs to know what step the agent is on, what artifacts exist, what failed, and whether the worker is stuck.

Its runtime follows a ReAct-like pattern. A base agent stores messages in memory, tracks state, runs step by step, enforces a maximum step count, and detects repeated/stuck behavior. The tool-calling agent asks the LLM for the next action, parses tool calls, executes them through a ToolCollection, appends observations back into memory, and stops when a special termination tool fires.

browser-use / OpenClaw

Browser-use and OpenClaw-style systems are browser/computer-use harnesses. Their defining feature is that the model's world is not a clean programmatic API. The world is a browser or GUI surface. The harness must convert pixels, DOM state, page metadata, tabs, URLs, screenshots, and recent events into a prompt the model can act on.

Therefore, the biggest change is in the actions this harness can take. Instead of "edit this file" or "run this command," the primitive actions are things like navigate, click, type, scroll, switch tabs, go back, upload a file, extract content, search the page, save a PDF, and wait. In browser-use, these actions are registered through a tool registry and compiled into an action model for the LLM. The model chooses structured actions; the harness executes them against a browser session and returns an ActionResult.

The context construction problem for such a harness is harder than it first appears. A screenshot is useful but often has parts you don't care about. A DOM tree is precise but often too large and not always aligned with what the user sees. An accessibility tree may capture interactive structure but miss visual layout. A robust browser harness, therefore, has to combine these signals into a bounded state summary: current URL, title, tabs, visible DOM, screenshot, scroll position, browser errors, pending network requests, and previous interactions.

The runtime for such a harness too has different needs. Given the lack of neat APIs, actions may not always work out. For example, Clicks can miss. Pages can navigate slowly. Popups can appear. Login walls can block progress. Elements can move. The harness therefore needs action timeouts, failure counts, loop detection, step limits, page extraction helpers, and often a judge or evaluator that decides whether the previous goal succeeded. If you've ever written a serious web-scraper, you can appreciate how much these little details matter.

Hermes Agent

Hermes is closer to a general agent runtime than a single-purpose product. Its center of gravity is not one environment like a repo or browser. It is a registry of tools, toolsets, skills, memory, browser/terminal integrations, cron jobs, delegation, and plugin hooks. This makes it useful for understanding harnesses as platforms.

The tool layer is registry-first. Tool modules self-register schemas, handlers, metadata, toolset membership, availability checks, and result limits. A separate model-tools layer discovers those tools, filters them by enabled or disabled toolsets, defers large tool catalogs behind search/describe/call bridge tools, dispatches function calls, applies middleware, runs pre-tool and post-tool hooks, and canonicalizes tool results before they return to the model.

This architecture exposes a different harness problem: tool abundance. If a runtime can connect many built-in tools, MCP tools, plugin tools, and skill-provided tools, dumping every schema into the model context is wasteful and confusing. Hermes-style systems need tool search, tool descriptions, scoped toolsets, availability checks, and lazy discovery.

Furthermore, Memory also takes center stage given the diversity of experiences. Hermes has file-backed curated memory with separate stores for agent memory and user memory. Memory is loaded into the system prompt at session start, while mid-session writes update disk but do not mutate the already-cached system prompt. There are many such harness-level tradeoff between persistence, safety, and prompt-cache stability.

The product and extensibility surface here is the star of the show. Skills, cron jobs, browser tools, terminal tools, file tools, voice tools, MCP OAuth, delegation, tool middleware, approval systems, and TUI state serve as the runtime. In this category, the core question is not "can the model complete this one task?" It is "how many kinds of tasks can this runtime safely support?"

A Worked out example comparing different harnesses

Suppose a user asks: "Fix the failing authentication tests."

In a local coding harness, the system first constructs context: current directory, repository instructions, git state, recent files, maybe previous memory about testing conventions. The model decides to inspect the test failure. It calls a shell tool to run the relevant tests. The runtime streams the command output back into the conversation. The model reads files, searches for authentication logic, edits code, reruns tests, and stops when the result is clean. Permissions may gate commands or edits. The product surface shows progress, diffs, and final summary.

In a remote worker harness, the same request may happen inside a cloud workspace. The agent may clone the repo, install dependencies, run tests, open a browser, persist logs, and expose a task timeline. If the session disconnects, it should resume. If tests take a long time, the user should be able to come back later. The model is still making decisions, but the harness is carrying the task across time.

In a browser harness, "fix the failing authentication tests" might instead mean operating a web dashboard or CI system: open the page, inspect logs, click into a failing run, copy an error, navigate to GitHub, and perhaps trigger a rerun.

In a general agent runtime, the same workflow might be built as a graph or agent program. One node reads CI status, another searches code, another proposes a patch, another asks for approval, another opens a pull request.

The model may be similar in all four cases. The harness is not, which results in entirely different products.

The zoom-out: Can this all be summed up as tool-use?

It is tempting to say all of these systems are "basically tool calls orchestrated by a model." That is functionally right, but incomplete in sentiment. The reality is that this process has significant design decisions baked in, which can often be even more important than the tools themselves. Therefore, the hard questions are not only "which tools exist?", but also:

• What does the model know before it chooses a tool?
• What state is hidden from the model?
• What information is retrieved from memory?
• What actions require user approval?
• How are failed actions represented back to the model?
• When does the loop stop?
• How does the system recover after a bad edit, wrong click, or broken command?
• What can be resumed tomorrow?

These decisions can influence an agent's behavior just as much, if not more, as the model weights. A model with poor context construction will act confidently on the wrong world. A model with powerful tools but weak permissions structure is dangerous. A model with memory but no retrieval discipline will remember irrelevant things and miss important ones. A model with no runtime limits will loop forever. A model with no product surface will be impossible to trust.

Learnings

The main learning is that we have put a lot of faith into the model itself, and conversations center around the newest best model. Better models clearly matter. They reason better, write better code, follow instructions better, and recover from mistakes more often. But it is important to remember that most useful applications of LLMs have not come from LLMs in isolation, but rather by building harnesses around them which allow models to do tasks in the real world.

A raw model does not have a filesystem. It does not have a browser. It does not know which files matter, which commands are safe, what state should persist, when to ask for approval, how to recover from a bad edit, or how to resume tomorrow. All of that are NOT attributes we have been able to teach an LLM. So, instead, we have built all of this around, and then taught LLMs how interact with this surrounding software.

Thus, making LLMs useful in the real world, is much more than training the next best LLM. Instead, it is what now would be called legacy or ordinary software engineering: tool design, permissioning, context construction, memory policy, runtime control, error handling, UI, observability, plugins, sandboxing, and recovery. Needless to say, it is therefore quite unclear how, if ever, LLMs could usher in the end of software engineering.