Experience from building general-purpose clouds can create limiting assumptions. True innovation in AI infrastructure requires questioning the established "best practices" of the past decade, recognizing that the core principles that made legacy clouds successful may not apply to this new, specialized world.
Mature AI applications are not static calls to a single large model. They are complex systems of many models that require a continuous "AI loop": tracing performance, identifying areas for improvement (cost, speed, accuracy), and constantly iterating by swapping models, fine-tuning, or refining prompts.
The future is multi-cloud. Instead of creating proprietary lock-in, the winning strategy is to embrace open standards and build tooling that can run anywhere. The "lock-in" comes from making your own managed infrastructure so performant and easy to use that customers choose to stay out of preference, not necessity.
The high cost of GPUs means any inefficiency during model training is extremely expensive. This economic reality justifies building specialized, AI-focused infrastructure with features like advanced observability and optimized storage to maximize GPU utilization and prevent costly delays from failures or slowdowns.
Simply adding GPUs to existing cloud infrastructure is insufficient for AI. An AI-centric approach requires re-imagining everything from hardware layout to job orchestration. Forcing AI concepts into legacy application models slows down development and creates unnecessary challenges for teams.
Unlike general-purpose cloud resources, AI training infrastructure with specialized networking (e.g., InfiniBand) and storage cannot be added fungibly. It requires significant pre-planning and deep integration, breaking the standard cloud deployment model of simply adding more commoditized compute or storage as needed.
Beyond catastrophic hardware failures, a significant problem in large-scale AI training is the "soft failure" of a single GPU slowing down. In a job using thousands of GPUs, identifying this one straggler is incredibly difficult without specialized observability tools built specifically for AI workloads.
The enterprise shift to AI will mirror the earlier shift to cloud, but happen twice as fast. Every large company will soon have a dedicated AI engineering team that will become the organization's primary focus. Traditional cloud infrastructure teams will shift into a supporting role for these new AI-centric initiatives.
The future of complex technical work, like AI model development, won't involve manually analyzing line charts in a GUI. Instead, developers and researchers will interact with AI agents conversationally, directing them to run experiments, analyze results, and suggest next steps, making today's dashboards obsolete.
SUNK combines two worlds: Slurm, the dominant job scheduler in high-performance computing and AI research, and Kubernetes, the standard for resilient, cloud-native infrastructure orchestration. This fusion gives researchers the powerful scheduling they need, backed by the robust, automated management they lack.