OpenAI's team found that as code generation speed approaches real-time, the new constraint is the human capacity to verify correctness. The challenge shifts from creating code to reviewing and testing the massive output to ensure it's bug-free and meets requirements.

Contrary to expectations of falling AI costs, the move from simple chatbots to complex, multi-step agentic systems is causing an explosion in token usage. A single user can trigger hundreds of agents, making expensive frontier models economically unsustainable for many application-layer companies.

Models that generate "chain-of-thought" text before providing an answer are powerful but slow and computationally expensive. For tuned business workflows, the latency from waiting for these extra reasoning tokens is a major, often overlooked, drawback that impacts user experience and increases costs.

Analysis of models' hidden 'chain of thought' reveals the emergence of a unique internal dialect. This language is compressed, uses non-standard grammar, and contains bizarre phrases that are already difficult for humans to interpret, complicating safety monitoring and raising concerns about future incomprehensibility.

In ultra-long tasks (e.g., 100 million tokens), AIs rely on summarizing previous context. This "compaction" process is lossy and can drop critical nuances. Once a flawed summary is made, the model tends to latch onto it, leading to significant errors and derailment, as seen in the UKAC incident.

Despite models advertising million-token context windows, Blitzy's CEO claims effective intelligence rapidly depreciates beyond 100k tokens due to "context pressure." This suggests that solving large-scale problems requires complex system-level orchestration, not just bigger models.

Classifying a model as "reasoning" based on a chain-of-thought step is no longer useful. With massive differences in token efficiency, a so-called "reasoning" model can be faster and cheaper than a "non-reasoning" one for a given task. The focus is shifting to a continuous spectrum of capability versus overall cost.

Despite massive context windows in new models, AI agents still suffer from a form of 'memory leak' where accuracy degrades and irrelevant information from past interactions bleeds into current tasks. Power users manually delete old conversations to maintain performance, suggesting the issue is a core architectural challenge, not just a matter of context size.

Contrary to the idea that infrastructure problems get commoditized, AI inference is growing more complex. This is driven by three factors: (1) increasing model scale (multi-trillion parameters), (2) greater diversity in model architectures and hardware, and (3) the shift to agentic systems that require managing long-lived, unpredictable state.