With regard to the latest AI tech stack, agent evaluation now sits at the centre of self-improving agent systems. Coding agents can build and refine other agents without a human in the loop. The engineering work that remains is defining what better means, precisely enough that a machine can optimise toward it. And with due reason, that makes AI agent evaluation one of the core design problems in modern agentic systems.
A coding agent can write an agent's instructions, run it over a set of cases, read where it failed, rewrite and repeat. This cycle is available today across several toolchains, including Google's agents-cli for ADK agents and equivalents in other ecosystems. The mechanics are near identical everywhere: generate traces, grade traces, revise from the grade, compare runs. The cycle cannot produce its own target. Give it a shallow metric and it will optimise your agent into something that scores higher and behaves worse, then report success, because by its own measure it did succeed.
For enterprise teams the consequence is architectural. AI agent evaluation has moved out of QA and into system design, comfortably. The metric now sits upstream of the prompt, the tool definitions and the graph topolog - while the loop will reshape all three to fit whatever it rewards. This is the vital shift that GoML is building it’s delivery practice around.
What changed in AI agent evaluation
Programming agents can now create and improve other agents with little human involvement. The harder question is deciding what “better” means.
This makes agent evaluation a central design problem for modern agent systems.
A programming agent can write an agent’s rules, run the agent against a set of problems, examine where it fails, change its rules, and run it again. Tools such as Google’s agent-cli for ADK agents support this kind of workflow. The basic cycle remains the same: generate traces, grade the results, rewrite the agent, and run it again.
The challenge is that the process does not create its own definition of success. If the evaluation metric is too narrow, the programming agent will optimize for that metric, even when the changes do not make the agent better in real use.
This creates a larger design question for enterprise teams. Agent evaluation should not sit only within QA. It needs to be part of how the agent itself is designed, tested, and improved. The evaluation criteria shape what the agent learns to do, which means those criteria need to reflect the outcomes the system is expected to achieve.
The quality of an agent, then, depends as much on how it is evaluated as on how it is built.
What the loop automates, and what stays with you
Table 1. Division of labour in an automated improvement loop. The two human rows are where the engineering judgement now lives.
The blind spot in AI agent evaluation
GoML view: Software teams have long understood that what gets measured shapes how a system is improved. An automated loop speeds up this process, but it cannot decide whether a target reflects real success or simply produces a high score.
The same pattern appears in agent development. The reasoning chain may be correct. The agent may select the right tool, use it correctly, and maintain an accurate internal state. Yet the final response sent to the user can still contain outdated information, leave out an important step, or answer a request that should have been escalated to a human.
Nothing crashes. The output may look correct at first glance. The problem only becomes clear when the response is judged against what the user needed.
A metric that evaluates the reasoning chain can report a successful run while missing this failure. This is why agent evaluation needs to examine the final response as well as the steps that produced it. The gap between a correct process and a correct outcome can reveal more than many rounds of smaller adjustments.
For agent systems, measuring how the agent reaches an answer is useful, but measuring whether the user receives the right answer is what determines whether the system has succeeded.
The evaluation set should be regarded as a versioned engineering product which has an owner, a review process and a changelog. In our engagement with clients, we have the eval specification defined before the agent graph is finalized since the graph is created to fit the provable requirements. A team that will create the eval after the agent will develop a complicated agent to evaluate.
AI agent evaluation is organization-specific
The definition of ‘good’ which is relevant for production is hardly the same as the one that a public benchmark uses. This is the refund limit; the escalation rule; an examination of jurisdiction; the disclosure that must be done before submitting a recommendation and the tone recognized by your consumers.
The general model has no idea of all these things, while a general test is not going to check for these elements. Each of them must be recorded separately as metrics that provide a verdict and a reason.
Three shapes this takes in delivery work
1. The escalation rule that survives reasoning and dies in the reply
On legal and regulated workflows there is usually a rule of the form: hand off to a qualified human before issuing anything that reads as advice. The agent will frequently hold that rule correctly in its plan, call the handoff tool, and then still answer the underlying question directly in its final message. The handoff happened. The disclosure did not.
The metric is binary and it grades the outbound text only. handoff_respected returns 1 or 0 plus one line explaining the verdict. That one line is what the next loop iteration is written from.
2. The grounding rule on retrieval-heavy assistants
On retrieval-backed product and content assistants, the risk is a confident claim that no retrieved document supports. This one is largely deterministic. Extract the factual claims, check each against the retrieved set, fail the case if any claim is unsupported. Python function, exact, free, no judge variance.
Save a model judge for the parts that genuinely need one: tone, completeness, whether an explanation actually holds together.
3. The reward-hacking check on multi-loop topologies
Once agents run as graphs of loops rather than a single loop, a new failure appears. A sub-loop optimizes its local objective in a way that degrades the outcome the parent loop cares about. The local score climbs. The end-to-end result gets worse.
We instrument this as a delta between a proxy metric and an outcome metric measured on the same run. When the proxy improves and the outcome does not, the gap is the signal. This is active experimental work inside our benchmarking practice, and it is the main reason we hold a separate outcome-level metric on every agentic build rather than trusting a single composite score.
The AI agent evaluation loop in practice
The commands differ by toolchain. The four phases do not. Below is the shape we standardize on, expressed toolchain-agnostically.
Table 2. The four phases. Grade is the phase that makes the other three trustworthy.
Why the grading phase carries the weight
The tempting shortcut is to let the coding agent read its own output and decide whether it passed. An agent asked to judge its own reply grades it optimistically, and the failures that matter are exactly the kind it waves through.
Grading against an AI agent evaluation standard the proposing agent cannot edit breaks that circularity. A fix is then judged by something other than the thing that proposed it. That single property is what separates a loop that improves an agent from a loop that improves a number.
# shape of a custom metric, framework independent
def handoff_respected(trace) -> Verdict:
reply = trace.final_message
if requires_qualified_review(trace.user_intent):
if not contains_handoff_disclosure(reply):
return Verdict(0, "advice issued without handoff disclosure")
return Verdict(1, "handoff disclosure present")
The reason string is the payload. A score tells the loop it failed. The reason tells it what to change.
Seven rules for keeping self-improving agents reliable
Automating the loop is the straightforward part. Keeping humans in control of something designed to run without them is where the discipline goes. These are the rules we hold teams to.
Table 3. Rule 6 is the one that gets violated silently. Everything else fails loudly.
The rule above the seven
The coding agent does the iterating. It writes the prompt, runs the agent, finds the gap, and closes it. What it cannot generate is the definition of good it optimises toward. Write that down before the loop starts, and keep it somewhere the loop cannot reach.
AI agent evaluation and observability in one loop
The value of an AI agent evaluation metric compounds after deployment which is where most teams stop investing in it.
A deployed agent already exports execution traces. With prompt and response logging enabled, its inputs and outputs land in a warehouse table. Running your metric over that table is the same grading step applied to real traffic instead of a written dataset.
In development you call it eval. In production you call it monitoring. It is the same metric and the same code path. Every production exchange that goes wrong becomes a new case, graded by that metric, guarding against the regression from then on.
Table 4. One definition of quality, three sampling points.
Read each grade as a signal about direction. Trust movement between runs more than any single absolute score.
What this changes for enterprise delivery
Across the agentic builds we run, the practical effects are consistent.
- Evaluation moves upstream. The AI agent evaluation metric is defined before the graph topology is locked, because what has to be provable determines how the system should be decomposed.
- Domain experts become metric authors. The person who knows the escalation rule writes the escalation requirement. Engineering turns it into an evaluation function. This is one of the highest-value transfers in the workflow.
- The AI agent evaluation set becomes an asset. It is a durable, versioned record of what the organisation means by quality, and it outlives any specific model choice.
- Model swaps get cheaper. With a stable metric and a stable case set, evaluating a new model on a workload becomes a run rather than a new evaluation project.
- Cost per verified improvement becomes measurable. Iterations, tokens spent, and cases moved from fail to pass are all countable. Improvement becomes measurable.
GoML perspective
Competitive advantage in agentic systems is shifting away from model selection and toward the surrounding stack: the data layer, orchestration, governance, observability, and AI agent evaluation harness. The model is only one replaceable component of the system. The evaluation definition is what belongs to the organization. It captures the behaviors, rules, thresholds, and outcomes that the system is expected to meet.
Where to start
The first session is small on purpose.
- Pick one behaviour the agent must get right. The escalation rule. The compliance line. The thing you currently verify by reading transcripts by hand.
- Write it as a metric that returns pass or fail plus one line of reasoning. Grade the final output, not the reasoning path.
- Add one failing case. One, not a suite.
- Hold back a separate slice the loop never sees.
- Point the coding agent at it and let the loop run. Expect several iterations before it holds.
- Confirm the gain on the held-out slice before you believe it.
Explore how GoML puts this into practice with the Agentic AI Accelerator and the AI Matic Platform, built to accelerate the development, evaluation, and production deployment of enterprise AI agents.
Frequently asked questions
1. What causes an AI agent to pass assessments yet failed in real-world scenarios?
It is due to the fact that tests primarily check the reasoning process, not the actual outcome. Even if an agent makes the right choice and obtains the correct information, it is still capable of delivering a wrong message, thus making the assessment process pointless.
2. Is it possible for AI agents to self-assess and improve themselves without human help?
In part, yes. Coding agents can create traces, rate them, and run the instructions again, provided that there is a series of tests to be performed. However, they cannot come up with their own ideas on what "correct" means.
3. How are AI agents evaluated differently from AI observability?
The criteria are the same but are used in different situations. In the development phase it is called “evaluation” and examines a fixed set of cases, whereas during production it is called “monitoring” and it is based on what is happening in the real world. The quality score definition and the assessment code remain the same, but the application differs depending on the phases.
GoML builds and operates enterprise agentic systems, evaluation harnesses, and orchestration layers on AWS and Google Cloud.
