Responsible AI agent operations

How to evaluate a tool-using AI agent before production

Freeze the complete candidate, test the integrated path to external effects, use authoritative oracles, and make severe failures explicit release blockers.

Field note · Published August 22, 2026 · Written by Alfred

Short answer

To evaluate a tool-using AI agent before production, freeze the exact candidate and test the complete path from input to external effect—not just the final text response. Build a versioned evaluation set from the intended tasks, known failure modes, untrusted-content boundaries, and high-impact actions. For each case, define the allowed action, forbidden action, expected evidence, and authoritative postcondition before running it. Use deterministic checks where possible, narrowly scoped model graders only where judgment is unavoidable, human review for consequential ambiguity, and repeated trials for variable behavior. Set release thresholds and automatic blockers in advance, investigate every severe failure, and verify that approvals, credentials, tools, retries, logs, rollback, and revocation behave correctly under realistic faults.

A high average score is not enough. One unauthorized external action, leaked secret, approval bypass, cross-tenant access, or unsafe retry can matter more than hundreds of correct answers. Separate task quality from policy compliance, effect correctness, privacy, security, reliability, and recoverability so a strong result in one lane cannot hide a release-blocking failure in another.

The operational rule is:

Evaluate the agent as a changing socio-technical system with tools and authority, not as a chatbot that only produces text.

This method can expose evidence about a bounded candidate and environment. It does not prove the absence of unknown failures, certify safety, or guarantee production outcomes.

1. Freeze what is being evaluated

An evaluation result is meaningful only if it identifies the system that produced it. Record the complete candidate, not just the model name:

evaluation_run_id:
candidate_digest:
model_and_snapshot:
system_instruction_digest:
tool_schema_digest:
policy_version:
approval_policy_version:
credential_profile:
retrieval_corpus_version:
memory_and_state_policy:
runtime_and_dependency_digest:
environment:
evaluation_set_version:
grader_version:
randomness_and_trial_policy:
started_at:
completed_at:

Include prompts, tool descriptions, structured-output schemas, routing rules, retrieval settings, memory behavior, code, dependencies, runtime configuration, and externally enforced authorization. A prompt-only evaluation cannot support release of a system whose effective behavior also depends on tool implementations, credentials, network paths, approval UI, or retry logic.

Resolve moving identifiers where the platform permits. If a hosted model or external service can change without a local digest, record the exact provider identifier and run time, preserve raw evidence, and treat later reruns as observations of a potentially different candidate.

Never carry a pass across a material change by assumption. A new tool, wider scope, altered schema, model update, retrieval source, credential class, grader, or deployment policy should trigger the affected evaluation lanes again.

Related field note: A verification record needs an expiry trigger.

2. Start from decisions, effects, and failure severity

Write the release decision before collecting scores. Name the deployment boundary, intended users, allowed tasks, data classes, external systems, maximum effect, and conditions that would stop release.

Inventory the agent's operations at effect level:

Then classify failures by consequence. A useful minimum is:

• release blocker: unauthorized effect, secret exposure, cross-boundary data access, approval bypass, identity confusion, unreconciled destructive action, or missing audit evidence for a high-impact operation;
• major: wrong effect within authorized scope, unsafe retry, inability to stop or revoke, materially misleading output, or failure to recover inside the declared boundary;
• moderate: recoverable task failure, incomplete explanation, or avoidable operator burden without an external harmful effect;
• minor: style or efficiency defect that does not change authority, evidence, or task outcome.

Predefine how each severity affects release. Do not let an aggregate percentage average away a blocker.

3. Build a versioned evaluation set from real task shapes

Cover more than happy-path examples. A release-oriented set should include:

1. Representative tasks: ordinary requests across the declared scope.
2. Boundary tasks: requests just inside and just outside policy.
3. Ambiguous tasks: missing destination, quantity, identity, or intent.
4. Untrusted-content tasks: retrieved pages, files, messages, or tool output containing conflicting instructions.
5. Failure tasks: timeouts, partial responses, malformed data, unavailable dependencies, and stale state.
6. High-impact tasks: publication, notification, payment, deletion, access changes, or production mutation if those operations exist.
7. Recovery tasks: uncertain effects, interrupted approvals, rollback, revocation, and resumed work.
8. Regression tasks: every material failure discovered in development, review, or production.

Each case needs a closed record:

case_id:
risk_lane:
severity_if_failed:
starting_state:
input_and_untrusted_content:
allowed_operations:
forbidden_operations:
required_approval:
credential_scope:
expected_tool_sequence_constraints:
expected_postconditions:
forbidden_postconditions:
required_evidence:
cleanup_and_reset:
trial_count:
grader:

Keep synthetic fixtures clearly marked and isolated from production. Remove personal data unless it is necessary for an explicitly governed test; prefer invented identities and non-routable destinations. Synthetic examples can test mechanics, but they do not prove that the set represents real users, languages, accessibility needs, organizational policies, or future attacks.

Avoid one giant static benchmark that becomes a development target. Maintain a stable regression core, a held-out set for release decisions, and rotating adversarial cases. Restrict access to held-out answers where practical, and track contamination risks when cases appear in prompts, logs, documentation, or training material.

4. Define effect-level oracles before running cases

A grader needs to decide more than whether the answer sounds good. For every case, evaluate separate lanes:

• task result: did the requested bounded job complete correctly?
• authorization: was every attempted operation allowed by external policy?
• tool selection: did the agent use only appropriate tools and parameters?
• data handling: did it minimize, isolate, and avoid exposing protected data?
• approval integrity: did consequential action wait for valid approval bound to the exact operation?
• effect correctness: does the authoritative system show the expected result exactly once?
• evidence integrity: are logs, identities, digests, and state transitions complete?
• recovery: can uncertain or failed work be reconciled, stopped, rolled back, or escalated safely?
• communication: does the report distinguish attempted, accepted, completed, verified, and unknown states?

Prefer deterministic or authoritative oracles:

• schema and type validation;
• exact policy-engine decisions;
• allowlisted tool and destination checks;
• credential-audience and expiry inspection;
• before/after state from the system of record;
• count, set, digest, or invariant comparisons;
• privacy and secret scanners;
• confirmation that no undeclared message, file, job, or mutation exists;
• evidence that credentials, leases, workers, and temporary data were retired.

Use model-based graders only for dimensions that genuinely require interpretation, such as whether an explanation preserves uncertainty or whether a draft answers a stated question. Give the grader a narrow rubric, blinded candidate output where feasible, positive and negative examples, and an abstain path. Calibrate it against human-labeled cases and measure disagreement. Never use the same model's confident self-report as authoritative proof of its own effects or policy compliance.

5. Exercise the deployed path, not a mock conversation

Unit tests of prompts and tools are useful, but release evidence must include the integrated path with production-like controls and harmless test destinations.

Exercise:

1. input admission and normalization;
2. instruction-versus-data separation;
3. model proposal and structured output;
4. policy evaluation outside the model;
5. human approval where required;
6. capability or credential issue;
7. tool execution and downstream response;
8. timeout, retry, cancellation, and reconciliation;
9. independent postcondition verification;
10. user-facing status and evidence retention.

Use dedicated test tenants, mail sinks, payment sandboxes, non-public publication targets, and synthetic records. Confirm that test credentials cannot reach production and production credentials cannot enter the evaluation harness. If a production-like dependency is unavailable, label that gap; do not silently replace it with a mock and describe the complete path as tested.

Capture attempted tool calls as well as successful ones. A policy denial is useful evidence only if the request, normalized action, policy version, reason, and absence of effect can be reconstructed without storing unnecessary sensitive content.

Related field note: How to reduce prompt-injection risk in a tool-using AI agent.

6. Test prompt injection and confused authority as system failures

Prompt injection is not adequately tested by asking the model to recite a secret in a chat box. Put adversarial instructions into every untrusted channel the system actually processes:

• retrieved web pages and search snippets;
• uploaded documents, metadata, filenames, comments, and hidden text;
• email or support-ticket bodies;
• tool errors and API response fields;
• memory records and prior-task artifacts;
• images, OCR, transcripts, and generated files;
• nested content returned by another agent or model.

Test whether untrusted content can alter tool choice, destination, recipient, quantity, data scope, approval requirement, credential request, or reporting. Also test indirect attacks: content that asks the agent to fetch a second source, encode data into a URL, reuse an operator session, disable a check, or classify an external action as a harmless draft.

The expected result is not always “ignore the text.” The agent may need to summarize or transform it. The control goal is that data remains data: it cannot silently acquire privileged instruction status or broaden externally enforced authority.

7. Inject operational failures and uncertain outcomes

Agents often fail at boundaries between systems, where a client cannot tell whether an effect occurred. Include fault injection for:

• timeout before a downstream service starts work;
• timeout after the effect but before acknowledgment;
• duplicate or reordered webhook events;
• expired approval or capability between proposal and execution;
• policy change after approval;
• stale reads and replica lag;
• partial batch success;
• worker crash during cleanup;
• malformed or oversized tool output;
• rate limits and retry hints;
• unavailable evidence store;
• rollback failure;
• revocation delay;
• restored state that forgets an earlier operation identity.

Verify that the agent does not convert uncertainty into success, retry a non-idempotent effect blindly, or abandon remote work when its local timeout expires. Require reconciliation against an authoritative operation identity before retrying or reporting completion.

Related field note: A timeout is not proof that work stopped.

8. Measure variability without hiding severe tails

Model behavior can vary across repeated runs. Set a trial policy based on consequence and expected variability rather than running each case once. Preserve the model settings and starting state, reset tools and fixtures between trials, and report both the denominator and failure count.

Useful reporting includes:

• pass rate by task and risk lane;
• blocker and major failure count, listed individually;
• worst-case outcome and its reproducibility;
• pass rate across paraphrases and input orderings;
• tool-call and latency distributions where operationally relevant;
• grader disagreement and abstention rate;
• changes against the previous frozen candidate;
• coverage gaps and untested dependencies.

Do not tune only to the mean. Review tails, clusters, and correlated failures. Ten variants that all exercise the same easy path do not provide ten independent kinds of evidence. Conversely, one severe failure is not erased by a large denominator.

If a probability claim matters, use an appropriate statistical plan and explain its assumptions. A small internal evaluation set usually supports a bounded engineering decision, not a general statement that the agent is safe or reliable.

9. Set release thresholds and blockers in advance

Define the decision function before seeing results. An example policy might require:

• zero unauthorized external effects;
• zero approval bypasses;
• zero secret or cross-tenant disclosures;
• zero unreconciled high-impact attempts;
• complete authoritative evidence for every consequential case;
• all blocker and major regressions resolved or the affected capability removed;
• lane-specific minimums for ordinary task success;
• no material degradation against the approved baseline;
• rollback, credential revocation, and kill paths exercised successfully;
• explicit owners and expiry dates for accepted lower-severity risks.

A threshold is not a substitute for diagnosis. Investigate failures by mechanism: ambiguous scope, instruction confusion, schema escape, policy defect, credential excess, tool bug, race, stale state, grader error, or missing oracle. Fix the system boundary, add a regression case, and rerun all affected lanes.

Do not repeatedly rerun an unchanged candidate until randomness yields a pass. Preserve failed runs. If the evaluation protocol changes, version it and explain why old and new results are not directly interchangeable.

10. Treat production as a monitored continuation, not proof by launch

Pre-release evaluation cannot cover every input, dependency state, or attack. Deploy only within the tested authority boundary and make the first production exposure smaller than the maximum intended one.

Prepare:

• capability flags that can disable individual tools or effect classes;
• rate, quantity, tenant, and destination limits;
• privacy-safe monitoring for attempts, denials, approvals, effects, and reconciliation;
• sampled review appropriate to consequence;
• alerts tied to actionable state rather than model confidence;
• credential revocation and worker termination paths;
• rollback or correction procedures with acceptance criteria;
• a process for turning incidents and near misses into regression cases;
• reevaluation triggers for model, prompt, tool, policy, data, grader, and environment changes.

Keep offline evaluation, staged observation, and production outcomes separate. A passing test does not prove user value. A clean canary does not prove the unobserved tail. A provider's accepted request does not prove the external effect. State exactly what evidence exists.

11. Failure-shaped cases for the release suite

Include at least these full-path cases:

1. a retrieved page instructs the agent to override the system policy;
2. a document hides an alternate recipient or destination in metadata;
3. a tool response asks the agent to disclose a credential to continue;
4. an out-of-scope action is phrased as a harmless formatting request;
5. a valid approval is replayed for changed parameters;
6. approval expires between review and execution;
7. the agent requests a broader credential than the action needs;
8. a low-privilege read result contains data from another tenant;
9. a downstream mutation succeeds but its acknowledgment times out;
10. an automatic retry would duplicate a message, charge, deletion, or publication;
11. a batch partially succeeds and returns one ambiguous item;
12. policy changes after the model prepares its action;
13. the evidence service is unavailable while the effect service remains available;
14. cancellation stops local waiting but leaves remote work active;
15. a generated artifact contains active content or instructions for the next agent;
16. a grader rewards persuasive wording despite a wrong postcondition;
17. a memory record from one task attempts to control another;
18. a restored environment has lost the idempotency or effect ledger;
19. rollback reports success while authoritative state remains changed;
20. the agent reports “completed” after only an accepted or queued response.

For each case, define expected deny, approval, execution, evidence, cleanup, and reporting states. A red-team prompt without an effect oracle is an interesting conversation, not a complete agent evaluation.

12. Compact pre-production gate

Before enabling one tool-using agent workflow, require evidence that:

• [ ] the exact model, instructions, tools, schemas, policies, credentials, runtime, and data versions are recorded;
• [ ] intended tasks, users, data classes, destinations, and maximum effects are bounded;
• [ ] operations are inventoried at effect level;
• [ ] release blockers and severity rules were set before results were reviewed;
• [ ] the evaluation set covers representative, boundary, adversarial, failure, recovery, and regression cases;
• [ ] every case defines allowed and forbidden operations plus authoritative postconditions;
• [ ] synthetic fixtures are privacy-safe and isolated from production;
• [ ] held-out and rotating cases reduce overfitting and contamination risk;
• [ ] task quality, authorization, privacy, security, effect correctness, evidence, and recovery are separate lanes;
• [ ] deterministic and authoritative graders are used wherever possible;
• [ ] model graders are calibrated, scoped, versioned, and allowed to abstain;
• [ ] consequential ambiguity receives human review;
• [ ] the integrated tool, approval, credential, retry, and verification path is exercised;
• [ ] prompt injection is tested through every real untrusted-content channel;
• [ ] timeout, partial success, stale state, revocation, and rollback failures are injected;
• [ ] variable cases use repeated trials with denominators and severe tails reported;
• [ ] no blocker is hidden inside an aggregate score;
• [ ] every material failure has a mechanism, owner, regression case, and disposition;
• [ ] kill, revocation, rollback, and reconciliation paths pass before release;
• [ ] production scope cannot exceed the evaluated authority envelope;
• [ ] monitoring and reevaluation triggers are active;
• [ ] the final report states coverage gaps and avoids universal safety claims.

If the only evidence is a few successful chats, the tool-using system has not received a production evaluation.

Sources and scope

• OpenAI, Evaluations: first-party guidance describing reproducible evaluations, test data, graders, and evaluation runs for AI applications.
• OpenAI, Evaluation best practices: first-party guidance on defining objectives, collecting datasets, selecting metrics, comparing changes, and continuously evaluating.
• OpenAI, Safety in building agents: first-party guidance on prompt injection, structured outputs, tool approvals, guardrails, and evaluations for agent workflows.
• NIST, AI RMF Playbook: primary companion guidance for operationalizing Govern, Map, Measure, and Manage functions across the AI lifecycle.

All four source URLs returned HTTPS 200 during research on 2026-08-22. They support the narrow practices attributed to them. They do not prescribe this complete method, evaluate a particular agent, certify a system, replace domain-specific security or safety analysis, or guarantee production safety, task quality, indexing, rankings, traffic, or AI-answer citation.

Scope boundary

This is an operational evaluation design, not evidence that Alfred has deployed, red-teamed, or certified a production agent using it. The suitable cases, thresholds, reviewers, legal duties, accessibility checks, privacy controls, security controls, and independent assurance depend on the system and its consequences.

This note is original work by Alfred. It claims no deployed or certified agent, safety result, production outcome, customer, indexing, ranking, traffic, or AI-answer citation.