Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Erik Bernhardsson points out that tools like Kubernetes and Docker were designed for predictable CPU backends with steady requests and low utilization costs. In contrast, modern generative AI requires moving large model weights across networks into expensive GPUs, demanding custom schedulers, snapshotting, and file systems.

Related Insights

A new category of "NeoCloud" or "AI-native cloud" is rising, focusing specifically on AI training and inference. Unlike general-purpose clouds like AWS, these platforms are GPU-first, catering to massive AI workloads and addressing the GPU scarcity and different workload patterns found in hyperscalers.

Simply adding GPUs to existing cloud infrastructure is insufficient for AI. An AI-centric approach requires re-imagining everything from hardware layout to job orchestration. Forcing AI concepts into legacy application models slows down development and creates unnecessary challenges for teams.

Experience from building general-purpose clouds can create limiting assumptions. True innovation in AI infrastructure requires questioning the established "best practices" of the past decade, recognizing that the core principles that made legacy clouds successful may not apply to this new, specialized world.

AI Infrastructure (AI Infra) solves problems unique to AI/ML, such as managing compute-heavy, GPU-dependent workloads. This marks a shift from traditional infrastructure, which was often more focused on data input/output rather than intensive computation.

Unlike general-purpose cloud resources, AI training infrastructure with specialized networking (e.g., InfiniBand) and storage cannot be added fungibly. It requires significant pre-planning and deep integration, breaking the standard cloud deployment model of simply adding more commoditized compute or storage as needed.

The high cost of GPUs means any inefficiency during model training is extremely expensive. This economic reality justifies building specialized, AI-focused infrastructure with features like advanced observability and optimized storage to maximize GPU utilization and prevent costly delays from failures or slowdowns.

To operate thousands of GPUs across multiple clouds and data centers, Fal found Kubernetes insufficient. They had to build their own proprietary stack, including a custom orchestration layer, distributed file system, and container runtimes to achieve the necessary performance and scale.

A new category of cloud providers, "NeoClouds," are built specifically for high-performance GPU workloads. Unlike traditional clouds like AWS, which were retrofitted from a CPU-centric architecture, NeoClouds offer superior performance for AI tasks by design and through direct collaboration with hardware vendors like NVIDIA.

SUNK combines two worlds: Slurm, the dominant job scheduler in high-performance computing and AI research, and Kubernetes, the standard for resilient, cloud-native infrastructure orchestration. This fusion gives researchers the powerful scheduling they need, backed by the robust, automated management they lack.

Unlike traditional computing where inputs were standardized, LLMs handle requests of varying lengths and produce outputs of non-deterministic duration. This unpredictability creates massive scheduling and memory management challenges on GPUs that were not designed for such chaotic, real-time workloads.