An AI agent successfully identified the origin of a 15-year-old Firefox bug by semantically tracing it through file renames and code moves, using advanced Git commands that a human expert didn't even know existed. This is a task that is exceptionally tedious for humans.

Don't ask an LLM to perform initial error analysis; it lacks the product context to spot subtle failures. Instead, have a human expert write detailed, freeform notes ("open codes"). Then, leverage an LLM's strength in synthesis to automatically categorize those hundreds of human-written notes into actionable failure themes ("axial codes").

To overcome the challenge of reviewing AI-generated code, have different LLMs like Claude and Codex review the code. Then, use a "peer review" prompt that forces the primary LLM to defend its choices or fix the issues raised by its "peers." This adversarial process catches more bugs and improves overall code quality.

Using one LLM to evaluate another's output ("LLM as a Judge") is a common but deceptively difficult technique. Chip Huyen highlights that companies can spend up to 80% of their development time just writing and refining the complex evaluation guidelines for the judge LLM.

Prompting a different LLM model to review code generated by the first one provides a powerful, non-defensive critique. This "second opinion" can rapidly identify architectural issues, bugs, and alternative approaches without the human ego involved in traditional code reviews.

LLMs can both generate code analysis tools (measuring metrics like cognitive complexity) and then act on those results. This creates a powerful, objective feedback loop where you can instruct an LLM to refactor code specifically to improve a quantifiable metric, then validate the improvement afterward.

While a powerful model like Mythos was helpful, the real breakthrough came from a custom-built 'harness' that gave the AI specific tools and integrated it into Mozilla's existing bug-fixing pipeline, turning raw model output into verified, actionable reports.

A one-size-fits-all evaluation method is inefficient. Use simple code for deterministic checks like word count. Leverage an LLM-as-a-judge for subjective qualities like tone. Reserve costly human evaluation for ambiguous cases flagged by the LLM or for validating new features.

You don't need to create an automated "LLM as a judge" for every potential failure. Many issues discovered during error analysis can be fixed with a simple prompt adjustment. Reserve the effort of building robust, automated evals for the 4-7 most persistent and critical failure modes that prompt changes alone cannot solve.