An AI model's operating environment—its "harness"—is now the primary driver of capability. Benchmarks show the same model achieves vastly different results in different harnesses, proving that the runtime, tools, and state management are as critical as the model's internal weights for achieving results.

While AI inference can be decentralized, training the most powerful models demands extreme centralization of compute. The necessity for high-bandwidth, low-latency communication between GPUs means the best models are trained by concentrating hardware in the smallest possible physical space, a direct contradiction to decentralized ideals.

When running AI inference at extreme scale, the most surprising and difficult challenges are often not unique to LLMs. Instead, they are classic distributed systems problems—like kernel panics caused by logging overload—that only manifest under immense load. The immaturity of runtimes exacerbates these issues.

Beyond catastrophic hardware failures, a significant problem in large-scale AI training is the "soft failure" of a single GPU slowing down. In a job using thousands of GPUs, identifying this one straggler is incredibly difficult without specialized observability tools built specifically for AI workloads.

AI performance engineer Chris Fregley warns that developing on local machines or even consumer-grade GPUs is a waste of time. Critical differences in hardware, memory bandwidth, and drivers mean that accurate profiling and optimization can only be done on the exact production systems, like NVIDIA's Blackwell or Hopper GPUs.

Optimizing AI systems on consumer-grade (e.g., RTX) or small-scale professional GPUs is a mistake. The hardware profiles, memory bandwidth, and software components are too different from production systems like Blackwell or Hopper. For performance engineering, the development environment must perfectly mirror the deployment target.

When splitting jobs across thousands of GPUs, inconsistent communication times (jitter) create bottlenecks, forcing the use of fewer GPUs. A network with predictable, uniform latency enables far greater parallelization and overall cluster efficiency, making it more important than raw 'hero number' bandwidth.

While the idea of distributed compute pools is appealing, it's not feasible for AI training due to high latency demands; GPUs must be physically co-located. However, AI inference is less sensitive to this lag, making a distributed network of compute (like home GPUs) a much more viable and exciting model.

Setting an LLM's temperature to zero should make its output deterministic, but it doesn't in practice. This is because floating-point number additions, when parallelized across GPUs, are non-associative. The order in which batched operations complete creates tiny variations, preventing true determinism.