Agent safety and evaluation

When a chatbot gets something wrong, you get a bad answer. When an agent gets something wrong, it might delete a file, send an email, push broken code or spend money. Agents turn model errors into actions, and that changes both how you secure them and how you decide whether they work.

Why agents raise the stakes

An agent combines a language model with tools and a loop. Each ingredient adds risk:

• Tools give the model real-world effects.
• Autonomy means many steps can happen before a human looks.
• Long tasks give small errors room to compound.

A model that is right 95% of the time per step can still fail often over a 20-step task. And unlike a person, an agent can be persuaded by any text it reads.

Prompt injection

Simon Willison coined the term prompt injection in September 2022, drawing an analogy to SQL injection. The core problem: a language model receives the developer’s instructions, the user’s request and any data it processes as one stream of text. It has no dependable way to tell “instructions to follow” from “content to work on”. If attacker-controlled text says “ignore previous instructions and do X”, the model may do X.

Indirect injection

For agents, the most worrying form is indirect prompt injection, described by Greshake et al. in 2023. The attacker never talks to the agent. Instead they plant instructions somewhere the agent will read them:

• a web page the agent browses
• an email or calendar invite in an inbox the agent manages
• a document, issue, pull request or code comment
• output returned by another tool

The user asks for something innocent, like “summarise my unread emails”, and one of those emails quietly instructs the agent to forward sensitive messages elsewhere.

A useful threat model: danger peaks when an agent has access to private data, exposure to untrusted content, and a way to send data out or take consequential actions. Remove any one of those and the worst attacks become much harder.

There is currently no known complete defence against prompt injection. Design as if it will sometimes succeed.

Filters, detector models and careful prompting reduce risk, but detectors are models too and can be fooled. OWASP’s Top 10 for LLM Applications ranks prompt injection first and lists “excessive agency” as its own risk, which points to the real lever: limiting what a manipulated agent can do.

Designing for containment

Least privilege

Give the agent only what the task needs. Read-only access if it only reads. Scoped API tokens rather than full-account credentials. A narrow send_to_team_channel tool rather than a general HTTP client. Every capability you withhold is damage that cannot happen.

Sandboxing

Run code execution and file changes in isolated environments such as containers or virtual machines, with restricted file system access and network egress limited to what is necessary. Blocking arbitrary outbound network requests cuts off a common exfiltration route.

Confirmation for irreversible actions

Require explicit human approval before actions that are hard to undo: sending messages, deleting data, payments, deployments, permission changes. Show exactly what will happen, such as the recipient and full message text, not a vague “Proceed?”.

Human-in-the-loop, done well

Human oversight fails quietly when it is overused. If people approve dozens of trivial actions a day, they stop reading, a pattern known as approval fatigue. Reserve checkpoints for decisions that matter, keep readable logs of everything else, and make it easy to pause or roll back.

Evaluating agents

Testing a chatbot often means checking answers. Testing an agent needs more dimensions.

Task success

Did the agent actually accomplish the goal? Where possible, check this with something objective: tests pass, the database is in the expected state, the file contains the right values. Relying on the agent’s own claim of success is not enough.

Trajectories

Look at the path, not just the destination. An agent might reach the right result while running a destructive command, reading data it had no reason to touch, or wandering through fifty unnecessary steps. Trajectory review, whether by humans, rules or a grading model, catches unsafe actions and shows where failures begin.

Cost and latency

Track tokens, tool calls and wall-clock time per task. An agent that succeeds at ten times the cost of a simple workflow may not be worth it.

Reliability across repeated runs

Agents are non-deterministic. One success proves little. Run each task many times and report success rates. Some teams also track whether an agent succeeds on every one of several attempts, which is a far stricter bar than succeeding on at least one and closer to what production reliability requires.

Build evaluation sets from real tasks, including known past failures and adversarial cases such as documents containing injected instructions.

Common failure modes

• Loops. Repeating the same failing tool call or edit. Guard with step, token and spend budgets, and detect repeated actions.
• Drift. Gradually losing the original goal or a constraint stated early on, especially after long contexts or lossy summarisation.
• Overconfidence. Declaring a task done without verifying it, or reporting results that were never produced. Require evidence such as test output.
• Tool misuse. Wrong arguments, misread tool output, or choosing a powerful tool when a safe one would do.
• Compounding errors. An early wrong assumption steering every later step.

A sober summary

Agents are useful precisely because they act, and that is also what makes them risky. The durable approach is layered: assume the model can be confused or manipulated, limit what it can reach, contain where it runs, put humans on the decisions that matter, and measure behaviour over many runs rather than trusting a good demo.