Instead of focusing on making Transformers cheaper, researchers should identify their inherent weaknesses. Jerry Tworek argues the current architectural bottleneck, not just scale or algorithms, is what's holding back progress toward smarter AI systems.
Intense market competition forces major AI labs to focus on scaling proven, profitable Transformer models for short-term gains. This creates a strategic blind spot, leaving a crucial gap for startups like Core Automation to explore fundamentally new, non-Transformer architectures that could redefine the field.
The core weakness of Transformers is their static nature. They are trained in a lab on a snapshot of data and then deployed. They cannot adapt to new events, tools, or user tasks without a full retraining cycle, making true continuous learning at test time impossible with the current architecture.
The choice of optimization algorithm dictates which model architectures are viable. While weaker optimizers require simpler models, a more powerful optimizer can successfully train more complex, harder-to-optimize architectures. This shows how optimizer and architecture research are deeply intertwined, unlocking new performance possibilities.
The key advantage of Transformers was their economic efficiency. The cost to train them was less than the revenue they could generate, making massive scaling investments justifiable. LSTMs, scaling less efficiently, would have been too expensive to train to a commercially impressive level, likely preventing the current AI boom.
AI development is inefficiently split into pre-training (optimizing for compression) and RL (optimizing for tasks), where RL often invalidates pre-training metrics. Combining these into a unified, end-to-end learning algorithm focused on final outcomes could yield an order-of-magnitude improvement in training efficiency.
Novel AI architectures are useless if they can't run efficiently on hardware. This requires custom GPU kernels, a task demanding rare expertise and creating a major bottleneck. Core Automation is focused on automating kernel generation to enable rapid architectural experimentation.
Jerry Tworek, a self-described "RL maximalist," found that scaling RL at OpenAI improved benchmarks but failed to solve real-world problems. The training data and evals were a closed loop, disconnected from the messy distribution of real user tasks, necessitating models that can learn at test time.
Chain-of-thought is a clever but inefficient "band-aid" for the Transformer's shallow architecture. It simulates deeper computation by generating more tokens, which is slow and expensive at inference. According to Core Automation's Rohan Anil, a superior architecture would have greater computational depth built-in.