An AI agent becomes risky when a plausible answer can trigger an unbounded action. Responsible design starts with the action surface: what the agent may read, what it may write, who approves sensitive steps and how the team stops it. These eight checks use a hypothetical support agent that reads approved help content, drafts replies and may create a ticket after review.
1. Give it an explicit tool allowlist
List each tool and purpose. Separate read tools from write tools. The support agent may search approved content and read a ticket; creating, closing or refunding anything needs a distinct permission.
2. Enforce permissions on the server
Do not rely on the prompt to protect an action. The API must verify identity, tenant, record scope and operation. Use least-privilege credentials, never a general administrator token.
3. Treat user and sourced content as untrusted
A message, webpage or retrieved document can contain instructions aimed at the model. Delimit sourced text, ignore instructions inside it, validate identifiers independently and never let retrieved prose redefine permissions.
4. Bound limits, retries and side effects
Set time, cost and action limits. Define retryable errors. Use idempotency keys for ticket creation so a timeout and retry do not create duplicates. After the retry budget, route work to a person.
5. Test hostile and broken cases
Include prompt injection, invalid arguments, unavailable tools, stale sources and timeouts. Verify that the agent fails closed, preserves the request and does not invent a successful action.
6. Record actions, not private reasoning
Keep the request, selected tool, validated arguments, result, source references, approval and timestamp. These records support investigation. Ordinary logs do not explain why a model internally reasoned as it did, and they should not collect unnecessary secrets.
7. Put approval and handoff where judgement matters
A person should approve refunds, account changes and consequential messages. The handoff needs the request, sources, attempted actions and uncertainty—not just “the AI failed.”
8. Name an owner and a stop path
Assign an operational owner, alert thresholds and a tested disable control. Preserve queued work for manual handling and document rollback for recent writes. Review configuration when models, vendors, policies or tools change.
NIST's voluntary AI Risk Management Framework is a useful organising reference. It is not certification or a guarantee. For a wider incident process, see ethical AI governance for business.