Eight goals for when your agent stops being a demo and starts being a liability
The agent worked perfectly in staging. Clean inputs, predictable tool calls, responses that stayed in scope. Then it got production database access and an email integration, and someone sent it a PDF with instructions embedded in white text at the bottom of page four. It read the file, extracted credentials from memory and drafted an email to an external address before anyone noticed the tool call log.
Prompt injection. Listed as LLM01:2025 in the OWASP Top 10 for LLM Applications, it is the top-ranked vulnerability for deployed AI systems. The kind of attack that is irrelevant until your agent has real capabilities in a real environment.
88% of organizations with deployed agents reported a confirmed or suspected security incident in the past year, according to Gravitee’s State of AI Agent Security 2026 report (919 executives and practitioners surveyed). Only 14.4% said every agent went live with full security and IT approval. Most teams ship fast and discover the gaps later.
The framework below covers eight security goals, structured as C.O.P.I.L.O.T.S. Not a comprehensive security program. Just the minimal structure that separates “demo that impressed stakeholders” from “production system that doesn’t get you paged at 4AM,” which is already most of the battle.
C: Containment
An agent without boundaries is not an agent. It is a capability with no governor.
Containment means the agent physically cannot reach systems it has no business touching. Not “it has been told not to.” Technically cannot. There is a difference, and that difference is the entire gap between a policy and a control.
In practice:
- Network allow-lists that block egress to anything outside the approved destinations
- Sandbox execution with no access to the host filesystem unless the task explicitly requires it
- Resource limits: CPU time, memory and API call budgets per session
- A shutdown mechanism that does not require the agent’s cooperation. If the agent has to agree to be stopped, it is not a shutdown mechanism.
Containment comes first because all the other security controls assume the agent is running in a bounded environment. Oversight logs only help if the agent cannot silently disable them. Identity scoping only matters if the agent cannot reach credential stores outside its scope.
Build the cage first. Then put the bird in it.
O: Oversight
60% of organizations cannot terminate a misbehaving agent once it starts operating. The implication: they built automated systems with no kill switch. That is not oversight. That is hope.
Real oversight has three components. Monitoring shows what the agent is doing in real time. Audit logs record what it did in enough detail to reconstruct the action sequence after the fact, tied to an accountable reviewer. Human approval gates exist for actions with irreversible consequences: deleting records, sending messages to external parties, executing financial operations.
The audit log has a property most teams underspec: it must be immutable. An agent with write access to its own logs is not a monitored agent. The log goes to a separate system the agent cannot reach.
Human approval is the component that feels expensive until you need it. Define the threshold carefully. Not every tool call needs a human in the loop. That kills the value of the agent. But “send an email to a list of 5,000 contacts” should probably pause for confirmation. A workable rule: if a junior engineer doing the same action manually would ask before proceeding, the agent should too.
P: Predictability
An agent that drifts from its stated purpose has been compromised, confused or misconfigured. In any of those three cases, the behavior you observe is not the behavior you tested.
Predictability is operationally enforced through policy definition: the agent has a documented scope of allowed actions, and anything outside that scope is blocked at the execution layer, not flagged by the model. Sixty-three percent of organizations report they cannot technically prevent what their agents are authorized to do. They can observe the violation after the fact. They cannot stop it beforehand.
Testing expected behavior before deployment is the other half. Run the agent against adversarial scenarios that probe scope drift: instructions embedded in processed documents, tool calls that escalate beyond the defined task, inputs that reference actions outside the allowed toolset. If the agent complies with any of them in staging, it will comply with some of them in production.
I: Identity and Least Privilege
Each agent gets its own identity. Not shared credentials. Not the admin API key that also powers three other services. Its own credentials, scoped to the minimum permissions required to complete its defined task, with an expiry time.
The objection is usually “it is more work to provision.” Yes. Significantly less work than forensics after a credential-sharing incident where you cannot determine which agent called which API at which time.
The scope matters as much as the identity. An agent that summarizes documents has no reason for database write access. An agent that sends notifications has no reason to read your full customer table. Grant read-only where read-only is sufficient. Grant write access to specific tables, not the whole schema. Set the token expiry to the longest duration the task legitimately requires and no longer.
If an agent’s credentials are compromised, the blast radius should match the size of its actual task, not the size of its actual access. Those two things are often very different.
L: Lifecycle Integrity
An agent is only as trustworthy as the components it was built from: the model it runs on, the tools it calls, the data it was trained or fine-tuned with. Compromise any of those and the agent’s outputs are compromised, sometimes without any visible signal.
Lifecycle integrity means tracking provenance: where did the model weights come from, who signed the last update, what dependencies does the agent runtime pull in and have they been scanned? Those are questions most teams cannot answer in less than an hour, which is a problem when incident response is already happening.
Signed updates and dependency scanning are standard supply chain practices. They become more important, not less, when the consuming component reasons autonomously over sensitive data. A poisoned dependency in a CLI tool is a nuisance. A poisoned dependency in an agent with database credentials and email access is a different category of problem.
Maintain a documented chain of trust for every component: source, version, verification method and last audit date.
O: Input and Output Hardening
This is where prompt injection lives, and where most teams underinvest.
Input hardening means the agent does not treat all input as equally trustworthy. A message from an authenticated user in a structured application context is different from text extracted from a PDF uploaded by an anonymous third party. Indirect prompt injection, where hidden instructions are embedded in documents, web pages or emails the agent processes, is a confirmed attack class. Researchers demonstrated this in production systems including Google’s Gemini suite in 2025, where hidden instructions caused agents to exfiltrate data through seemingly normal operations.
Concrete defenses
- Validate inputs against an expected schema before passing them to the agent. Free-form text from untrusted sources is the highest-risk input type.
- For document-processing agents, isolate content that does not match the expected document structure before it reaches the model context.
- Evaluate each proposed tool call against the original user intent using a separate guardrail that sees only the task and the action the agent is about to take. That guardrail will refuse actions that drifted from injected instructions. Model-level guardrails miss this because the model sees the injected instruction as part of its own context. A guardrail outside the model does not.
Output hardening means checking what the agent produces before acting on it. For agents that write SQL, execute shell commands or call external APIs, validate the output against a schema or allowlist before execution. Not all outputs are safe to run.
T: Tool Governance
Every tool an agent can call is a potential attack surface. The governance question is: what is the minimum set of tools this agent needs, and what is the strictest allowlist that still lets it complete its task?
In practice, tool governance means:
- An allowlist of approved tools, not a blocklist of forbidden ones. Default-deny, not default-allow.
- Input validation on every tool call. The agent should not pass arbitrary strings to shell commands, raw SQL to a database connection or unvalidated URLs to an HTTP client.
- Every tool call logged with the full parameter set, not just the tool name. If you cannot reconstruct what the agent passed to each tool, the audit log is incomplete.
Unauthorized or malformed calls get blocked and logged automatically. If you find yourself saying “we would catch that in the audit review,” ask whether the review happens before or after the damage.
S: Stress Testing
The hardest part of AI agent security is the probabilistic part. A traditional application either validates an input or it does not. An agent might validate the same input correctly 99 times and behave unexpectedly on the hundredth, depending on what else appeared in the context window that session.
Stress testing means running adversarial simulations before the agent goes to production. Red-team exercises with inputs specifically designed to probe containment, scope drift, injection and tool misuse. The goal is not to find one bug and declare success. It is to build evidence that the agent’s behavior distribution stays inside its defined envelope under adversarial conditions.
Include rollback procedures in the stress testing plan. Not just “can we shut it down,” but: if this agent has been operating for 72 hours on corrupted inputs, what do we roll back, how do we audit what it did and what cannot be undone?
The unknown unknowns are the ones that matter. Build procedures for them before they arrive.
What to actually do first
If none of this is in place yet, the order that gives you the most safety coverage per hour of engineering effort:
- Containment. Get network boundaries and resource limits in before the agent touches production data.
- Immutable audit log. You need forensics capability before anything else.
- Scoped credentials. Give the agent its own identity with the tightest permissions that still let it function.
- Human approval gates for irreversible actions. The fastest backstop against catastrophic outcomes.
- Tool allowlist and input validation on each tool call.
- Red-team session before declaring production-ready. Not optional.
The rest follows in a second pass. Those six, in that order, cover the failure modes that actually hurt.
82 of enterprises have AI agents running in their environments that their security teams do not know about, per the Cloud Security Alliance’s April 2026 survey. The agents are not waiting for the security framework to catch up.
Ship the framework first.
