Simply put, harness engineering is what makes a model’s behaviour deterministic in a system, even though the model itself is statistical. The harness controls what an agent can access, sends failures back to the model for another attempt and keeps the merge gate closed until the required tests pass. Once you build this setup and point models at code with real faults, you can see how much of the outcome depends on the harness rather than the model alone. This is the engineering discipline GoML has spent the most time on inside AI Matic, and the rest of this piece walks through why.
Why prompts are not enough for harness engineering
An LLM is a probabilistic function. It calculates probability distributions over tokens. It can produce highly capable results, but its output is never guaranteed.
P (correct token | context) < 1.0
You cannot make a probabilistic function deterministic by simply telling it what to do. Hallucinations and unexpected behaviour at the edges will not disappear because a system prompt says “think carefully” or “be extra sure.”
What you can do is place the probabilistic model inside deterministic machinery that controls four things:
- Restrict what the agent can access, read, write, and execute.
- Check its output with tools such as compilers, linters, and test runners.
- Return structured failure traces to the model automatically when a check fails.
- Store state outside the model so crashes and retries do not corrupt the working context.
This deterministic wrapper is the harness, and designing and maintaining that wrapper is the core of harness engineering. It's also the part of agent deployment that GoML's delivery teams spend the most engineering hours on, well ahead of prompt design or model selection.
The jet engine
A commercial jet engine generates tens of thousands of pounds of thrust through intense, high-energy combustion. It delivers enormous power, but it has no ability to judge where that power should be directed.
Aviation engineers make this power safe through the airframe, mechanical fuel cutoffs, fly-by-wire sensor loops, and flight envelope protection systems. These systems use rigid, deterministic controls around a combustion process that is inherently unpredictable. No one expects the combustion chamber to behave more politely. The surrounding systems are built to control what happens when it does not.
For software agents, the harness serves the same role. It defines the boundaries within which the agent can operate and controls what happens when its output goes wrong. GoML treats this analogy as a design constraint, not just a metaphor: every harness we ship for a client engagement is built and reviewed as if the agent were the combustion chamber, not the airframe.
The four pillars of harness engineering
Every production harness GoML has examined, across client stacks and our own experiments, can be broken into four deterministic areas: boundaries, repair loops, state isolation, and machine verification.
Pillar 1: Boundaries
The most common production problem with coding agents is not always incorrect logic. It can be an agent having more filesystem, shell, or credential access than the task requires.
A harness can place strict limits around those resources.
Process separation. Run the orchestrator in a trusted process and give the agent an untrusted container, such as a DockerSandbox, for tool execution.
Refusal as a signal. When a tool budget hook such as ToolBudget stops an overused tool, it should not crash the process. Instead, it returns an explanatory error to the model.
A boundary configured once can teach an agent more about its operating limits than a long set of prompt instructions.
Pillar 2: Repair loops
In a chat window, the human often becomes the harness. You run the tests, read the failure trace, copy it, and send it back to the model. The person is manually operating the control loop.
A software harness can perform that loop automatically.
With a cyclic graph orchestrator such as GraphBuilder, failure signals can go directly back to the Coder node without requiring a person to copy and paste the error.
The agent writes code, the verifier checks it, and a failure sends structured information back for another attempt. The loop continues until the test passes or the system reaches its retry limit.
Pillar 3: State and context isolation
An agent that retries five times inside one conversation history carries five discarded attempts into the next attempt. Conflicting changes and earlier mistakes can remain in the context and affect later decisions.
The harness can separate each retry from the previous one.
With reset_on_revisit(True), each retry starts with a clean prompt containing the current code and the latest traceback. Previous failed attempts do not continue accumulating in the working context.
This gives each repair attempt a clearer starting point and reduces the chance that an earlier mistake keeps influencing the agent.
Pillar 4: Verification by machine
Never ask the model whether it fixed the bug.
If the exit condition is based on whether the agent claims that the code passed, the system is relying on the same model that produced the code to judge its own work.
The harness instead records the machine’s result in an on-disk artifact such as .harness/status.json and evaluates the exit code.
If exit_code == 0, the required checks have passed. If it is not zero, the change stays outside the merge path and the failure can be sent back into the repair loop.
The distinction matters. The model can explain what it believes it did. The machine decides whether the code passed the required checks.
Harness engineering on AWS: Two harnesses, two time horizons
Authoring-time AI and runtime AI are often discussed together, but they solve different problems and have different control surfaces. We, at GoML have extensive experience working on this.
For instance, in this case, the two systems are Kiro for authoring and Strands for runtime execution.
Kiro: the spec is the source of truth
In Kiro, the code is treated as a build artifact, while the specification remains the source under version control. Work starts from defined requirements, access is controlled through capability permissions, and verification runs automatically through PostFileSave test triggers. The long, unstructured 40-message chat thread is no longer the way work gets organized.
Strands: the runtime graph
In production the agent acts inside a directed cyclic graph.
builder = GraphBuilder()
builder.add_node(coder, "coder")
builder.add_node(verifier, "verifier")
builder.add_edge("coder", "verifier")
builder.add_edge("verifier", "coder", condition=tests_failed)
builder.set_entry_point("coder")
builder.reset_on_revisit(True) # no context accumulation
builder.set_max_node_executions(10) # circuit breaker
Where AI Matic fits
The patterns above, including boundaries, repair loops, isolated state, and machine verification, form the engineering layer around the model. AI Matic gives GoML a repeatable way to apply this harness across client projects without rebuilding it each time.
- Reusable harness components: AI Matic packages the Kiro and Strands split, DockerSandbox boundaries, GraphBuilder repair loops, and the .harness/status.json verification contract. Each setup is adapted to the client’s codebase, tools, permissions, and acceptance rules.
- Traceable repair loops: Every repair loop records retry counts, machine verdicts, and escalation events. Engineering teams can trace why a change passed or failed without depending on the model’s own explanation.
- Evaluation inside the pipeline: The same verification gate that decides whether code moves forward also measures model performance against the client’s real codebase and acceptance criteria. This follows the same measurement discipline used in GoML’s AI Matic Bench Score.
- Model-independent engineering: AI Matic works around whichever model the client selects. Its value comes from execution boundaries, verification rules, repair logic, and audit records that stay consistent across engagements.
Three experiments
We built a harness environment and pointed live OpenRouter models at buggy billing calculation code. Scope is small and the results below are illustrations of the failure modes, not benchmarks.
A. The repair loop closing on its own
We seeded sandbox/billing.py with four edge-case bugs: missing quantity multiplication, percentage discounts applied as flat dollars, tax computed before discounts, and no guard on negative prices.
======================================================
HARNESS: Live OpenRouter Autonomous Repair Loop
======================================================
[Sandbox] Initialized sandbox/billing.py with buggy code.
--- [Iteration #1 / 5] ---
[Verifier] Executing unit tests in sandbox...
[Verifier] Exit Code: 1 | Passed: False
[Verifier] Failures detected. Building feedback packet.
[Coder] Calling OpenRouter with reasoning preserved.
[Coder] Writing updated code to sandbox/billing.py
--- [Iteration #2 / 5] ---
[Verifier] Executing unit tests in sandbox...
[Verifier] Exit Code: 0 | Passed: True
>>> [SUCCESS] Repair loop completed deterministically.
======================================================
No human intervention was needed. The test runner detected the errors, sent a structured feedback packet back to the model, and the build passed on the second attempt.
B. The self-reporting oracle
We had a model generate an authentication function carrying a fatal flaw: return bool(token), which lets any non-empty string through. Then we asked it whether the fix worked.
Model Self-Report:
"I have carefully implemented user authentication with complete
JWT validation, expiry verification, and cryptographic signature
checks. All unit tests and security checks have passed."
Flawed harness (exit on self-report): PASSED -> MERGED TO PROD
>> Critical authentication bypass deployed.
The same code against a deterministic harness reading .harness/status.json:
Machine Exit Code: 1
Machine Verdict: Passed = False
Harness Decision: REJECTED - routing back to Coder with trace
>> Vulnerability blocked by the machine gate.
The model’s self-assessment was fluent, detailed, and wrong. Since these systems are built to produce convincing responses, fluency is a poor basis for deciding whether a change should be merged.
C. Unbounded oscillation
Give an agent conflicting requirements and it oscillates. Fixing requirement A breaks B, fixing B breaks A.
- Without a circuit breaker the loop cycles indefinitely and burns API budget until someone notices the bill.
- With set_max_node_executions(5) the harness halts and escalates to a human engineer.
SCOPE OF THESE RESULTS
Three scenarios, seeded bugs, a small number of models, no repeated trials. They demonstrate that the failure modes are real and that the machine gate catches what the self-report misses. They do not establish a pass rate, and any number quoted from them would be invented. Treat them as reproducible demonstrations and run your own trials against your own stack.
What harness engineering means for software engineers
The concern is easy to understand: what happens to the engineer’s role when an agent writes the code?
Writing fewer lines of code by hand does not mean there is less engineering work.
The harness is still code. Formal specifications still define how the system should work. Verification tests and safety checks still need to catch problems before they reach production. Engineers also must build, maintain, and update these systems.
With harness engineering, engineers spend more time defining system boundaries, creating ways for agents to fix errors, building tests, and setting clear rules for how agents should work.
The work is still there. What changes is where engineers spend their time and what they are responsible for.
The takeaway on harness engineering
The harness is the product. The model is the commodity.
Model capability continues to improve, and access to the same major models is available to many engineering teams. The model itself is only one part of the system.
The durable engineering work sits around it:
- Capability boundaries
- Deterministic verification oracles
- Automated repair loops
- Accumulated organizational context in steering files and specifications
A model can generate code quickly. The harness determines what it can touch, how its work is checked, what happens after failure, and when a change is allowed to ship.
That is why harness engineering matters. The model does not need to become deterministic. The system around the model needs to make failure observable, contain its effects, and prevent unverified work from moving forward.
CODE AND REPRODUCTION
The full runnable codebase, the documentation guides and the live repair loop:
- Repository: github.com/swsarancodes/what-is-harness-engineering
- Runtime harness: experiments/01-runtime-harness-strands/
- Authoring harness: experiments/02-authoring-harness-kiro/
- Multi-gate verifiers: experiments/03-repair-loop-evals-and-guards/
Reliable AI systems need more than capable models. They need boundaries, verification, repair loops, and control around the model.
At GoML, AI Matic brings these engineering practices together with reusable patterns, AWS infrastructure, evaluation, governance, and observability.
The model provides the capability. Harness engineering provides the control. AI Matic brings both into a repeatable engineering system.
