Get your free personalized podcast brief
We scan new podcasts and send you the top 5 insights daily.
Unlike competitors, MatX's ML team conducts fundamental research, training LLMs to validate novel hardware choices. This allows them to safely "cut corners" on industry standards, such as using less precise rounding methods. This deep co-design of model and hardware creates a uniquely efficient product.
Related Insights
Existing AI chips force a trade-off: high-throughput HBM memory (NVIDIA, Google) has high latency, while low-latency SRAM memory (Grok) has poor throughput. MatX's architecture combines both, putting model weights in fast SRAM and inference data in high-capacity HBM to achieve both low latency and high throughput.
Startups can make big bets on emerging workloads, like LLMs before they were proven. This is a product risk. In contrast, incumbents like Google or NVIDIA must ensure their next chip serves a wide range of existing customers, forcing them to be more conservative and avoid disruptive product bets.
Designing custom AI hardware is a long-term bet. Google's TPU team co-designs chips with ML researchers to anticipate future needs. They aim to build hardware for the models that will be prominent 2-6 years from now, sometimes embedding speculative features that could provide massive speedups if research trends evolve as predicted.
NVIDIA's commitment to CUDA's backward compatibility prevents it from making fundamental changes to its chip architecture. This creates an opportunity for new players like MatX to build chips from a blank slate, optimized purely for modern LLM workloads without being tied to a decade-old programming model.
Top-tier kernels like FlashAttention are co-designed with specific hardware (e.g., H100). This tight coupling makes waiting for future GPUs an impractical strategy. The competitive edge comes from maximizing the performance of available hardware now, even if it means rewriting kernels for each new generation.
Designing a chip is not a monolithic problem that a single AI model like an LLM can solve. It requires a hybrid approach. While LLMs excel at language and code-related stages, other components like physical layout are large-scale optimization problems best solved by specialized graph-based reinforcement learning agents.
Model architecture decisions directly impact inference performance. AI company Zyphra pre-selects target hardware and then chooses model parameters—such as a hidden dimension with many powers of two—to align with how GPUs split up workloads, maximizing efficiency from day one.
True co-design between AI models and chips is currently impossible due to an "asymmetric design cycle." AI models evolve much faster than chips can be designed. By using AI to drastically speed up chip design, it becomes possible to create a virtuous cycle of co-evolution.
The current 2-3 year chip design cycle is a major bottleneck for AI progress, as hardware is always chasing outdated software needs. By using AI to slash this timeline, companies can enable a massive expansion of custom chips, optimizing performance for many at-scale software workloads.
At a massive scale, chip design economics flip. For a $1B training run, the potential efficiency savings on compute and inference can far exceed the ~$200M cost to develop a custom ASIC for that specific task. The bottleneck becomes chip production timelines, not money.
