Instead of manually crafting complex evaluation prompts, a more effective workflow is for a human to define the high-level criteria and red flags. Then, feed this guidance into a powerful LLM to generate the final, detailed, and robust prompt for the evaluation system, as AI is often better at prompt construction.
Related Insights
Instead of manually crafting a system prompt, feed an LLM multiple "golden conversation" examples. Then, ask the LLM to analyze these examples and generate a system prompt that would produce similar conversational flows. This reverses the typical prompt engineering process, letting the ideal output define the instructions.
Don't ask an LLM to perform initial error analysis; it lacks the product context to spot subtle failures. Instead, have a human expert write detailed, freeform notes ("open codes"). Then, leverage an LLM's strength in synthesis to automatically categorize those hundreds of human-written notes into actionable failure themes ("axial codes").
After deconstructing successful content into a playbook, build a master prompt. This prompt's function is to systematically interview you for the specific context, ideas, and details needed to generate new content that adheres to your proven, successful formula, effectively automating quality control.
Instead of manually refining a complex prompt, create a process where an AI agent evaluates its own output. By providing a framework for self-critique, including quantitative scores and qualitative reasoning, the AI can iteratively enhance its own system instructions and achieve a much stronger result.
Achieve higher-quality results by using an AI to first generate an outline or plan. Then, refine that plan with follow-up prompts before asking for the final execution. This course-corrects early and avoids wasted time on flawed one-shot outputs, ultimately saving time.
Product managers may lack the expertise to create comprehensive evals from scratch. A better approach is to generate initial outputs with a base model, have subject matter experts review them, and use their direct feedback to define what constitutes a failure. It's easier for experts to spot mistakes than to predict them.
A one-size-fits-all evaluation method is inefficient. Use simple code for deterministic checks like word count. Leverage an LLM-as-a-judge for subjective qualities like tone. Reserve costly human evaluation for ambiguous cases flagged by the LLM or for validating new features.
When a prompt yields poor results, use a meta-prompting technique. Feed the failing prompt back to the AI, describe the incorrect output, specify the desired outcome, and explicitly grant it permission to rewrite, add, or delete. The AI will then debug and improve its own instructions.
The prompts for your "LLM as a judge" evals function as a new form of PRD. They explicitly define the desired behavior, edge cases, and quality standards for your AI agent. Unlike static PRDs, these are living documents, derived from real user data and are constantly, automatically testing if the product meets its requirements.
The most effective way to build a powerful automation prompt is to interview a human expert, document their step-by-step process and decision criteria, and translate that knowledge directly into the AI's instructions. Don't invent; document and translate.