To combat 'token maxing,' Palantir created 'Evolve,' a tool that analyzes production logs to recommend optimal AI models and workflow changes. One customer used it to swap models, tune prompts, and re-architect to eliminate 60% of their token costs in just two days.

While swapping an API endpoint for a new AI model is trivial, the real barrier is the extensive QA and re-benching required. Each new model has qualitatively different outputs, necessitating a full product testing cycle to ensure it doesn't degrade user experience, creating high practical switching costs.

Sophisticated model routers do more than route queries to the cheapest AI model. Palantir's Evolve tool also automatically optimizes prompts for the target model, a dual approach that can reduce token consumption by 60% and overall compute costs by up to 97% for specific tasks.

PMs often default to the most powerful, expensive models. However, comprehensive evaluations can prove that a significantly cheaper or smaller model can achieve the desired quality for a specific task, drastically reducing operational costs. The evals provide the confidence to make this trade-off.

OpenAI favors "zero gradient" prompt optimization because serving thousands of unique, fine-tuned model snapshots is operationally very difficult. Prompt-based adjustments allow performance gains without the immense infrastructure burden, making it a more practical and scalable approach for both OpenAI and developers.

The improved quality from AI agent loops comes at a steep price. Anthropic engineers shared an example where a task that took 20 minutes and cost $9 with a simple prompt required 6 hours and $200 using an agent loop. This highlights the current cost-benefit trade-off for adopting this advanced technique.

The true cost of fine-tuning isn't the initial training but the ongoing maintenance. Base foundation models experience significant capability improvements every 2-3 months. This pace means a custom fine-tuned model can quickly fall behind, forcing a continuous and expensive re-tuning cycle.

OpenAI found that removing repeated instructions from old prompts improved scores by 10-15% while cutting token usage by 66%. The complex rule lists built for older models now confuse systems like GPT-5.6, leading to worse and more expensive answers.

Mature AI applications are not static calls to a single large model. They are complex systems of many models that require a continuous "AI loop": tracing performance, identifying areas for improvement (cost, speed, accuracy), and constantly iterating by swapping models, fine-tuning, or refining prompts.