Generative AI is predictive and imperfect, unable to self-correct. A 'guardian agent'—a separate AI system—is required to monitor, score, and rewrite content produced by other AIs to enforce brand, style, and compliance standards, creating a necessary system of checks and balances.

While guardrails in prompts are useful, a more effective step to prevent AI agents from hallucinating is careful model selection. For instance, using Google's Gemini models, which are noted to hallucinate less, provides a stronger foundational safety layer than relying solely on prompt engineering with more 'creative' models.

Contrary to the popular belief that generative AI is easily jailbroken, modern models now use multi-step reasoning chains. They unpack prompts, hydrate them with context before generation, and run checks after generation. This makes it significantly harder for users to accidentally or intentionally create harmful or brand-violating content.

While a general-purpose model like Llama can serve many businesses, their safety policies are unique. A company might want to block mentions of competitors or enforce industry-specific compliance—use cases model creators cannot pre-program. This highlights the need for a customizable safety layer separate from the base model.

AI model capabilities have outpaced their value delivery due to a fundamental design problem. Users are inherently scared and distrustful of autonomous agents. The key challenge is creating interaction patterns that build trust by providing the right level of oversight and feedback without being annoying—a problem of design, not technology.

AI's unpredictability requires more than just better models. Product teams must work with researchers on training data and specific evaluations for sensitive content. Simultaneously, the UI must clearly differentiate between original and AI-generated content to facilitate effective human oversight.

As AI models become more powerful, they pose a dual challenge for human-centered design. On one hand, bigger models can cause bigger, more complex problems. On the other, their improved ability to understand natural language makes them easier and faster to steer. The key is to develop guardrails at the same pace as the model's power.

Undersecretary Rogers warns against "safetyist" regulatory models for AI. She argues that attempting to code models to never produce offensive or edgy content fetters them, reduces their creative and useful capacity, and ultimately makes them less competitive globally, particularly against China.

Using a large language model to police another is computationally expensive, sometimes doubling inference costs and latency. Ali Khatri of Rinks calls this like "paying someone $1,000 to guard a $100 bill." This poor economic model, especially for video and audio, leads many companies to forgo robust safety measures, leaving them vulnerable.