The performance gains from Nvidia's Hopper to Blackwell GPUs come from increased size and power, not efficiency. This signals a potential scaling limit, creating an opportunity for radically new hardware primitives and neural network architectures beyond today's matrix-multiplication-centric models.

Despite using inferior chips due to export restrictions, DeepSeek achieved massive cost savings by discovering and utilizing underdocumented hardware features, such as bypassing a specific cache. This proves that deep hardware exploration can yield greater gains than simply acquiring more powerful GPUs.

The plateauing performance-per-watt of GPUs suggests that simply scaling current matrix multiplication-heavy architectures is unsustainable. This hardware limitation may necessitate research into new computational primitives and neural network designs built for large-scale distributed systems, not single devices.

The key metric for AI chips (GPUs/TPUs) is achieving a high percentage of theoretical peak performance (e.g., 70-80%). This concept, known as "mechanical sympathy," is largely absent in the CPU world, where software performance is so inefficient that measuring against peak is considered nonsensical.

Top-tier kernels like FlashAttention are co-designed with specific hardware (e.g., H100). This tight coupling makes waiting for future GPUs an impractical strategy. The competitive edge comes from maximizing the performance of available hardware now, even if it means rewriting kernels for each new generation.

Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.

While many focus on compute metrics like FLOPS, the primary bottleneck for large AI models is memory bandwidth—the speed of loading weights into the GPU. This single metric is a better indicator of real-world performance from one GPU generation to the next than raw compute power.

AI coding assistants struggle with deep kernel work (CUDA, PTX) because there's little public code to learn from. Furthermore, debugging AI-generated parallel code is extremely difficult because the developer lacks the original mental model, making it less efficient than writing it themselves.

Instead of using high-level compilers like Triton, elite programmers design algorithms based on specific hardware properties (e.g., AMD's MI300X). This bottom-up approach ensures the code fully exploits the hardware's strengths, a level of control often lost through abstractions like Triton.