Enterprises are currently overspending on tokens by sending all queries to the most powerful LLMs. A new software category will emerge to intelligently route requests to smaller, cheaper models when possible, creating a critical efficiency and cost-saving layer between companies and foundational model providers.

While often discussed for privacy, running models on-device eliminates API latency and costs. This allows for near-instant, high-volume processing for free, a key advantage over cloud-based AI services.

Voyage AI's "Matryoshka" embeddings structure vectors like Russian nesting dolls. A high-dimension vector (e.g., 1024) contains lower-dimension versions (e.g., 512). This allows developers to test performance vs. cost simply by truncating the vector, avoiding the lengthy process of re-embedding the entire dataset.

The optimal strategy for enterprise AI is not to rely solely on expensive frontier models. Instead, companies use a powerful model like Claude or GPT-4 to plan tasks and then delegate the execution to cheaper, fine-tuned open-source models. This massively reduces cost while maintaining high performance.

Instead of each team member paying for API access, Buzz allows one powerful machine to host a local LLM. The team can then connect to and share this single compute resource, making powerful AI more accessible and affordable for bootstrapped teams.

Embeddings from different sizes of Voyage AI models are compatible. This lets teams embed production data with a large model while developers use a free, local “Nano” model for queries. This novel approach eliminates token costs during development and testing, improving developer experience.

Relying solely on premium models like Claude Opus can lead to unsustainable API costs ($1M/year projected). The solution is a hybrid approach: use powerful cloud models for complex tasks and cheaper, locally-hosted open-source models for routine operations.

Criteo builds multiple, specialized foundation models (for products, user timelines, etc.) rather than a single monolithic one. The embeddings from these models are made available across the company, serving as a "warm start" to accelerate the development and improve the performance of new AI products.

Companies are building intelligent systems that analyze a user's prompt and automatically route it to the most cost-effective model that can handle the task. This avoids using expensive frontier models for simple requests, with some companies like Coinbase successfully keeping costs flat despite exponential usage growth.