The initial deployment of a new AI cluster sees a high failure rate, with 10-15% of new-generation GPUs like Blackwell needing to be returned or reseated. This "infant mortality" is a standard operational challenge for data centers, underscoring the physical difficulties of scaling AI infrastructure with bleeding-edge chips.

Publicly announcing the number of GPUs a lab possesses is "bravado" and a poor indicator of its actual power. True capability is measured by model output and performance, as compute utilization varies wildly. Focusing on inputs instead of outputs is a common mistake.

Top-tier data centers operate at extreme efficiency. Google's Borg team aimed for 96%+ node utilization, viewing anything less as a critical failure. This contrasts with MFU (Matrix Multiply Unit) utilization, where best-in-class is 60-70%. Most single-tenant clusters fall far short of Google's standards.

Beyond catastrophic hardware failures, a significant problem in large-scale AI training is the "soft failure" of a single GPU slowing down. In a job using thousands of GPUs, identifying this one straggler is incredibly difficult without specialized observability tools built specifically for AI workloads.

Google's AlphaChip team initially failed to impress the internal TPU team by optimizing for standard academic benchmarks. The breakthrough came when they co-developed cost functions with the TPU team that directly targeted the real-world metrics engineers were evaluated on, like congestion and power consumption.

Measuring 'tokens per watt' is insufficient. The key metric is 'tokens per completed, accepted task per watt.' This reframes efficiency as a yield problem, where tokens spent on rejected or useless outputs are like defective silicon, wasting both compute and power.

As the NeoCloud market matures, reliability has become the critical benchmark. Analysts now simulate hardware failures to test providers. Top players like CoreWeave can detect and recover in 20 minutes using hot spares, while lower-tier providers can take days, highlighting a massive operational gap that impacts customer trust.

When splitting jobs across thousands of GPUs, inconsistent communication times (jitter) create bottlenecks, forcing the use of fewer GPUs. A network with predictable, uniform latency enables far greater parallelization and overall cluster efficiency, making it more important than raw 'hero number' bandwidth.

A cheap model that fails often becomes expensive due to retries, fallbacks, and human review. The true measure of economic efficiency is the cost to reliably complete a task, not the raw inference cost, which can be a misleading metric at scale.