Tag: Production AI

A staging agent once “fixed” a support ticket by refunding the wrong order. The reply sounded confident. The tool call succeeded. The business outcome was still wrong.

That is why I care less about demo wow-factor and more about how we evaluate AI agents in production before they get real tools, real data, and real money paths. This post is for engineers and for engineering managers or business owners who need a plain-language checklist: what to measure, when to say no, and how to keep risk under control without drowning in jargon.

The short version

An agent is not a chatbot with extra steps. It is a small control loop:

User / system goal
        │
        ▼
   Plan the steps
        │
        ▼
 Call tools / APIs  ──►  (side effects: refund, email, ticket…)
        │
        ▼
 Read the evidence
        │
        ▼
 Answer or act  ──►  Success? Escalate? Stop?

If you only score the final English sentence, you are grading theatre. Production evaluation scores the loop: did we finish the job, with the right tools, on evidence we can trust, at a cost and risk we accept?

Who this is for (two audiences, one scoreboard)

Same system. Different questions. One shared dashboard beats three separate slide decks.

Four things I measure (and why)

1. Task success — “Case closed correctly” beats “sounded smart.”
2. Tool correctness — Calling the wrong refund API with a 200 OK is still a failure.
3. Honesty — If the tool said “not found,” the agent must not invent a tracking number.
4. Ops & risk — Slow, expensive, or thrashing agents are production incidents waiting to happen.

I will not open high-impact write tools (payments, access changes, bulk emails) until all four are visible on a dashboard someone actually watches.

A small metric set people will look at

*Targets are for a narrow agent (a few clear intents). A “do anything” bot will miss them — that is a scope problem, not a model brand problem.

Shadow mode before real side effects

This is the part non-technical stakeholders usually like, because it is easy to explain in a steering meeting.

SHADOW (safe)
  Agent proposes: “Call Refund(order=123, $40)”
        │
        ▼
  Policy check + log proposal
        │
        ▼
  Human / existing system still does the real work
        │
        ▼
  Compare proposal vs what a good operator would do

LIVE (earned later)
  Agent proposes → policy allows → system executes
        │
        ▼
  Still keep logs, limits, and a kill switch

Only after proposals match a human baseline for a boring stretch of time do I open live writes — and high-risk actions may still need confirmation or dual control.

Go / no-go gates (no multi-week manifesto)

I use five gates. Fail one → fix the root cause → re-measure. Do not “hope harder” with a bigger model first.

“The exec loved the demo” is not a gate.

What good traces look like (enough detail, not a science project)

When something goes wrong on a Thursday morning, you need a story you can open:

• What was the user trying to do? (redact personal data)
• Which model / prompt version ran?
• Which tools were available, and which were called?
• Did policy allow, deny, or escalate — and why?
• How long did it take, and what did it cost?
• Did we complete the goal, or hand off to a human?

Without that, teams argue opinions. With that, you can see whether the bug is the model, a flaky API, a bad tool description, or a missing business rule.

If your tools sit behind something like an MCP gateway, put allowlists and audit logs in that control plane — not only in the system prompt. Prompts are guidance. Gateways and policies are brakes.

Anti-patterns (in normal language)

1. Demo-driven production — shipping because it looked good once.
2. Safety by pep talk — “never refund unless…” with no hard allowlist.
3. Eighty tools on day one — the agent gets lost; start with a short menu.
4. Success = no error toast — HTTP 200 is not “correct business outcome.”
5. No kill switch — if you cannot turn it off quickly, you are not ready.
6. Metrics with no owner — a dashboard nobody owns is wallpaper.

What I would do first on a new agent

1. Pick 3–5 intents only (example: order status, draft refund under a cap, open a ticket).
2. Expose a small tool list with clear inputs and default-deny policy.
3. Log the six metric families above from day one.
4. Run offline tests for schema + “must refuse” cases.
5. Shadow writes until proposals are boringly good.
6. Canary a few percent of traffic with fast human backup.
7. Only then widen scope.

Multi-agent frameworks and fancy planners can wait. Measurement first.

FAQ

What does it mean to evaluate AI agents in production?

It means tracking real outcomes: job finished, tools used correctly, answers grounded in evidence, and cost/risk under control — on live or shadow traffic, not only on a static quiz.

We are not a technical team. What is the one slide we need?

Show task success, escalations, blocked/wrong tools, cost per success, and whether write actions are still in shadow mode. If those five are healthy, the rest is implementation detail.

Are offline tests enough?

No. They catch schema and policy holes early. Real phrasing, slow tools, and messy data only show up online. Use offline to fail fast; use shadow/canary to earn trust.

When is a bigger model the right fix?

After you have ruled out bad tool design, missing policy, and unclear scope. A larger model will not fix an agent that can call seventy overlapping APIs with no brakes.

How narrow should v1 be?

Narrow enough that you can list the intents on one slide and name one owner for failures. If the pitch is “it can do everything,” keep it out of production tools.

Closing

To evaluate AI agents in production, treat them like any other system that can change business state: define success, measure the loop, keep hard limits outside the prompt, and open power only after shadow mode gets boring.

Cool demos create interest. Quiet dashboards create trust. I would rather ship a small agent that is allowed to do three things well than a broad one nobody can explain when it fails.