To automate feedback and improve agent work from 50% to 90% completion, create a QA process where you instruct one agent to have its work reviewed by a 'panel' of other agents. This adversarial review loop identifies flaws and refines the output before human intervention.

Don't ask an LLM to perform initial error analysis; it lacks the product context to spot subtle failures. Instead, have a human expert write detailed, freeform notes ("open codes"). Then, leverage an LLM's strength in synthesis to automatically categorize those hundreds of human-written notes into actionable failure themes ("axial codes").

An AI agent reviewing its own code is prone to confirmation bias, as it operates from the same context that created an error. To achieve genuine quality assurance, use a different AI model, preferably from another vendor, for review. This introduces diverse training and uncovers blind spots.

Don't start building evaluations from a blank slate. Use an AI agent to analyze your production traces and automatically generate a baseline 'vibe eval.' This initial evaluation won't be perfect, but it provides a starting point for refinement and accelerates the improvement loop.

AI can generate rule-based "top-down" evaluations from a task description (e.g., word count). However, discovering nuanced "bottom-up" evaluations requires human intuition from reviewing many real-world data samples to find subtle, recurring failure modes.

As AI agents generate vast amounts of output, human review becomes an impossible bottleneck. The solution emerging is multi-agent systems where a separate 'grading agent' automatically scores and requests revisions on an agent's work against a predefined rubric, as seen in Anthropic's 'Outcomes' feature, enabling scalable quality assurance.

Instead of using generic tools like spreadsheets for error analysis, leverage an AI agent to build a custom HTML interface. The agent analyzes your data's structure and renders it with visual encodings that make it far easier for a human to review and spot issues.

Instead of seeking a "magical system" for AI quality, the most effective starting point is a manual process called error analysis. This involves spending a few hours reading through ~100 random user interactions, taking simple notes on failures, and then categorizing those notes to identify the most common problems.

Define different agents (e.g., Designer, Engineer, Executive) with unique instructions and perspectives, then task them with reviewing a document in parallel. This generates diverse, structured feedback that mimics a real-world team review, surfacing potential issues from multiple viewpoints simultaneously.