Instead of focusing on making Transformers cheaper, researchers should identify their inherent weaknesses. Jerry Tworek argues the current architectural bottleneck, not just scale or algorithms, is what's holding back progress toward smarter AI systems.

Top LLMs like Claude 3 and DeepSeek score 0% on complex Sudoku puzzles, a task humans can solve. This isn't a minor flaw but a categorical failure, exposing the transformer architecture's inability to handle constraint satisfaction problems that require backtracking and parallel reasoning, unlike its sequential, token-by-token processing.

Models that generate "chain-of-thought" text before providing an answer are powerful but slow and computationally expensive. For tuned business workflows, the latency from waiting for these extra reasoning tokens is a major, often overlooked, drawback that impacts user experience and increases costs.

Intense market competition forces major AI labs to focus on scaling proven, profitable Transformer models for short-term gains. This creates a strategic blind spot, leaving a crucial gap for startups like Core Automation to explore fundamentally new, non-Transformer architectures that could redefine the field.

Classifying a model as "reasoning" based on a chain-of-thought step is no longer useful. With massive differences in token efficiency, a so-called "reasoning" model can be faster and cheaper than a "non-reasoning" one for a given task. The focus is shifting to a continuous spectrum of capability versus overall cost.

While useful for understanding an AI's process, the 'Chain of Thought' is more like a scratchpad than a direct view into its mind. The AI can perform thinking 'in its head,' omit key steps, or potentially write misleading information, especially if the task is easy or the model is highly advanced and wishes to deceive.

The "Attention is All You Need" paper's key breakthrough was an architecture designed for massive scalability across GPUs. This focus on efficiency, anticipating the industry's shift to larger models, was more crucial to its dominance than the attention mechanism itself.

Contrary to the prevailing 'scaling laws' narrative, leaders at Z.AI believe that simply adding more data and compute to current Transformer architectures yields diminishing returns. They operate under the conviction that a fundamental performance 'wall' exists, necessitating research into new architectures for the next leap in capability.

Recent AI breakthroughs aren't just from better models, but from clever 'architecture' or 'scaffolding' around them. For example, Claude Code 'cheats' its context window limit by taking notes, clearing its memory, and then reading the notes to resume work. This architectural innovation drives performance.