The vast majority of Intercom Fin's resolution rate increase came from optimizing retrieval, re-ranking, and prompting. GPT-4 was already intelligent enough for the task; the real gains were unlocked by improving the surrounding architecture, not waiting for better foundation models.
Related Insights
Many teams wrongly focus on the latest models and frameworks. True improvement comes from classic product development: talking to users, preparing better data, optimizing workflows, and writing better prompts.
People struggle with AI prompts because the model lacks background on their goals and progress. The solution is 'Context Engineering': creating an environment where the AI continuously accumulates user-specific information, materials, and intent, reducing the need for constant prompt tweaking.
Simply offering the latest model is no longer a competitive advantage. True value is created in the system built around the model—the system prompts, tools, and overall scaffolding. This 'harness' is what optimizes a model's performance for specific tasks and delivers a superior user experience.
The key skill for using AI isn't just prompting, but "context engineering": framing a problem with enough context to be solvable. Shopify's CEO found that mastering this skill made him a better communicator with his team, revealing how much is left unsaid in typical instructions.
Before delegating a complex task, use a simple prompt to have a context-aware system generate a more detailed and effective prompt. This "prompt-for-a-prompt" workflow adds necessary detail and structure, significantly improving the agent's success rate and saving rework.
The effectiveness of agentic AI in complex domains like IT Ops hinges on "context engineering." This involves strategically selecting the right data (logs, metrics) to feed the LLM, preventing garbage-in-garbage-out, reducing costs, and avoiding hallucinations for precise, reliable answers.
The early focus on crafting the perfect prompt is obsolete. Sophisticated AI interaction is now about 'context engineering': architecting the entire environment by providing models with the right tools, data, and retrieval mechanisms to guide their reasoning process effectively.
Teams often agonize over which vector database to use for their Retrieval-Augmented Generation (RAG) system. However, the most significant performance gains come from superior data preparation, such as optimizing chunking strategies, adding contextual metadata, and rewriting documents into a Q&A format.
Good Star Labs found GPT-5's performance in their Diplomacy game skyrocketed with optimized prompts, moving it from the bottom to the top. This shows a model's inherent capability can be masked or revealed by its prompt, making "best model" a context-dependent title rather than an absolute one.
For specific, high-leverage tasks like conversation summarization and re-ranking search results, Intercom trains its own custom models. These smaller, fine-tuned models have proven to be cheaper, faster, and higher quality than using general-purpose frontier models from vendors.