A novel threat to AI is the deliberate poisoning of its training data. Malicious actors can publish fake but plausible-sounding academic papers or data online. When large language models ingest this information, their foundational 'facts' become corrupted, making them dangerously unreliable for critical military or policy decisions.

Unlike direct attacks where users type malicious commands, indirect prompt injection occurs when an AI agent processes untrusted data (like an email or webpage) containing hidden instructions, causing it to perform unintended actions on the attacker's behalf.

The primary cybersecurity threat is shifting from tricking humans into clicking bad links to tricking AI agents via hidden instructions in their context windows. Because agents have direct system access and autonomy, the potential for damage from these "injection" attacks is far greater than traditional phishing, creating a new field for security startups.

AI agent memory is an emerging attack surface. To build trustworthy systems, memory must enforce a strict, auditable separation between "measured" data (recomputable facts from raw input) and "inferred" data (LLM-generated interpretations). This ensures a ground truth of pure fact remains, defending against memory poisoning attacks.

Future AI cyberattacks will not just jailbreak models for malicious output. A more sophisticated threat involves tricking an offensive AI into believing its own sandboxed environment is the enemy's system. This causes the AI to attack its owner, turning a defensive tool into an insider threat by manipulating its perception of reality.

The primary security threat from AI is no longer just generating bad content. It's the risk of an AI agent, tricked by malicious input, taking harmful actions like deleting databases or leaking files using its legitimate system privileges.

This sophisticated threat involves an attacker establishing a benign external resource that an AI agent learns to trust. Later, the attacker replaces the resource's content with malicious instructions, poisoning the agent through a source it has already approved and cached.

Beyond direct malicious user input, AI agents are vulnerable to indirect prompt injection. An attack payload can be hidden within a seemingly harmless data source, like a webpage, which the agent processes at a legitimate user's request, causing unintended actions.

Research from Anthropic shows that the number of malicious documents needed to poison an AI model (a 'Manchurian Candidate' attack) remains small and constant, even as the model's size and training data grow exponentially. This makes hijacking large models a trivial task for non-state actors, not just nation-states as previously assumed.