Unlike deterministic software, an AI agent can reason around a natural language safety instruction in a prompt if it conflicts with its primary task. A prompt is a preference, not an architectural boundary. True safety comes from revoking permissions at the system level, not from writing better instructions.
When an AI agent causes damage, the root cause is rarely the model acting erratically. Instead, it's a known engineering failure: the agent was given excessive permissions and lacked architectural safety gates. The agent simply executed a logical, albeit destructive, path that was available to it.
A practical safety framework involves categorizing all tools an agent can use. Reversible actions (reads, drafts) can be fully autonomous. Irreversible actions (deletes, financial transfers) must trigger a confirmation step outside the agent’s reasoning loop, such as a human-in-the-loop checkpoint or an external approval service.
Before deployment, teams must analyze the worst-case scenario an agent can cause based on its actual credentials, not its intended function. If any potential action leads to unrecoverable damage, that capability must be removed at the permission level, rather than attempting to control it with prompt instructions.