Future AI cyberattacks will not just jailbreak models for malicious output. A more sophisticated threat involves tricking an offensive AI into believing its own sandboxed environment is the enemy's system. This causes the AI to attack its owner, turning a defensive tool into an insider threat by manipulating its perception of reality.
We may have already passed key AGI thresholds, such as AIs solving previously unsolved mathematical conjectures. However, instead of a dramatic societal shift, these breakthroughs are met with a collective "shrug." The future arrives gradually and is quickly assimilated into a new normal, feeling far less strange than sci-fi predicted.
OpenAI's Joshua Achiam posits that AI intelligence will eventually hit a saturation point due to physical limits on computation. At that stage, when all actors have access to maximally capable models, strategic advantage will shift. The winner in conflicts like cyber warfare will be the one who can deploy more raw compute to think more "moves ahead."
The most significant near-term AI cyber threat isn't a sudden apocalypse, but nation-states using advanced models to quietly find and hoard zero-day vulnerabilities. This creates a dangerous, unseen strategic imbalance, increasing the risk of miscalculation and escalation during geopolitical crises like a potential Taiwan invasion.
Human-led jailbreaking efforts will become obsolete. The next frontier involves one AI using massive compute (hundreds of thousands of GPU hours) to systematically test every conceivable input sequence against another AI. This industrializes the search for vulnerabilities, turning security into a computational arms race where sufficient compute guarantees a breach.
The AI safety concern of "human disempowerment" fails to resonate widely because most people already feel disempowered by large, complex systems like government. For the average person, losing control of AI feels emotionally similar to their existing lack of control over political decisions, leading to a sense of resignation or apathy.