The initial deployment of a new AI cluster sees a high failure rate, with 10-15% of new-generation GPUs like Blackwell needing to be returned or reseated. This "infant mortality" is a standard operational challenge for data centers, underscoring the physical difficulties of scaling AI infrastructure with bleeding-edge chips.

Top-tier data centers operate at extreme efficiency. Google's Borg team aimed for 96%+ node utilization, viewing anything less as a critical failure. This contrasts with MFU (Matrix Multiply Unit) utilization, where best-in-class is 60-70%. Most single-tenant clusters fall far short of Google's standards.

The high cost of GPUs means any inefficiency during model training is extremely expensive. This economic reality justifies building specialized, AI-focused infrastructure with features like advanced observability and optimized storage to maximize GPU utilization and prevent costly delays from failures or slowdowns.

When running AI inference at extreme scale, the most surprising and difficult challenges are often not unique to LLMs. Instead, they are classic distributed systems problems—like kernel panics caused by logging overload—that only manifest under immense load. The immaturity of runtimes exacerbates these issues.

Simply "scaling up" (adding more GPUs to one model instance) hits a performance ceiling due to hardware and algorithmic limits. True large-scale inference requires "scaling out" (duplicating instances), creating a new systems problem of managing and optimizing across a distributed fleet.

When splitting jobs across thousands of GPUs, inconsistent communication times (jitter) create bottlenecks, forcing the use of fewer GPUs. A network with predictable, uniform latency enables far greater parallelization and overall cluster efficiency, making it more important than raw 'hero number' bandwidth.

The report of XAI's low GPU utilization reveals a critical, non-obvious bottleneck in AI: it's not just about acquiring compute, but using it efficiently. This 'FLOPS utilization' problem, caused by architectural and load-balancing issues, means billions in hardware sits underused, creating an opportunity for companies that can optimize the compute stack.

In 2019, 99% of workloads used a single GPU, not because researchers lacked bigger problems, but because the tooling for multi-GPU training was too complex. PyTorch Lightning's success at Facebook AI demonstrated that simplifying the process could unlock massive, latent demand for scaled-up computation.

When building systems with hundreds of thousands of GPUs and millions of components, it's a statistical certainty that something is always broken. Therefore, hardware and software must be architected from the ground up to handle constant, inevitable failures while maintaining performance and service availability.