Legacy data centers were built with 20 to 30 year horizons to house flexible, low-power racks pulling 10 to 40 kilowatts. AI accelerator racks now draw hundreds of kilowatts to megawatts. Designing a facility to accommodate both low-power storage and high-power accelerators across decades leads to massive overbuilding. AI facilities must instead be purpose-built and co-designed directly with the power, cooling, and hardware requirements of the targeted workloads.
Theoretical FLOPS measure maximum compute potential in isolation, ignoring real-world operating friction. At scales of 100,000 accelerators, bleeding-edge components fail multiple times per day or hour. Because synchronous workloads halt entirely when a single node drops, progress depends on checkpointing, debugging, and recovery speed. Google evaluates performance using 'goodput'—the net useful work delivered toward the actual workload under real failure conditions rather than raw theoretical throughput.
While hardware performance continues to experience substantial year-over-year multiplier gains, Google observes that the majority of intelligence per watt gains come from model and software improvements. To double serving token capacity every six months, engineering teams rely on dozens or hundreds of stacked model-level and runtime optimizations to ensure underlying hardware is fully utilized, rather than depending solely on silicon advances.
Designing custom silicon offers substantial speed and power advantages, but hyper-specializing creates severe operational risk if workloads evolve over a chip's multi-year lifetime. For instance, Google's TPU v8i (inference) and v8t (training) are optimized for specific jobs but still capable of running the other. Retaining cross-workload fungibility ensures clusters do not become stranded assets if traffic projections shift over the hardware's six-year lifespan.
Building abstracted, cloud-agnostic AI stacks creates flexibility and eliminates vendor lock-in, but it introduces an impedance mismatch between software, runtime, networking, and silicon. Deep co-design across hardware and model architecture unlocks compounding 10% to 2x efficiency multipliers across every layer. Maximizing delivered intelligence per watt requires trading away some degree of fungibility in exchange for hardware-software co-optimization.
Interactive user chats provide multi-second human delays between queries, whereas deep horizon agentic workloads communicate autonomously in milliseconds. Because these agents constantly reason, fetch external state, and re-prompt models, demand shifts beyond raw accelerator compute into traditional infrastructure: orchestration CPUs, high-speed RAM, and storage. Data center topologies must adapt by managing the density and inter-building networking required between accelerator pods and CPU-heavy orchestration clusters.
Rather than converting optical signals into the electrical domain for traditional packet routing, Google employs optical circuit switches with 3D MEMS mirrors that redirect raw light. Within large TPU clusters, this architecture allows system controllers to route traffic around a failed accelerator rack directly to a hot spare rack in milliseconds without manual cabling changes or physical hardware interventions.
Supplying gigawatt-scale data center capacity off-grid with 99.99% reliability requires building 2x power capacity for one-plus-one redundancy, making isolated generation prohibitively expensive. Remaining connected to the public utility grid leverages statistical multiplexing across a broader consumer base. Google can utilize grid power during normal operation and feed local backup generation back to utilities during peak seasonal residential demand.
Frontier model training benefits from massive, geographically concentrated sites with tight network locality and uniform accelerator racks. Conversely, inference serving requires distributing smaller clusters globally to minimize latency and meet regional endpoints. Serving facilities also demand a tighter co-location of storage, CPU, and accelerators, preventing the extreme physical specialization seen in pure training clusters.
Terrestrial solar panels yield power roughly 30% of the day, demanding large battery backups. In a sun-synchronous orbit, data centers receive 1.4x solar intensity due to zero atmospheric attenuation and achieve 90% to 100% continuous sunlight coverage. This setup largely removes battery storage from orbital compute economics, though thermal dissipation and physical hardware servicing remain difficult technical hurdles.