Simply creating an LLM judge prompt isn't enough. Before deploying it, you must test its alignment with human judgment. Run the judge on your manually labeled data and analyze the results in a confusion matrix. This helps you see where it disagrees with you (false positives/negatives) so you can refine the prompt and build trust.

Effective enterprise AI deployment involves running human and AI workflows in parallel. When the AI fails, it generates a data point for fine-tuning. When the human fails, it becomes a training moment for the employee. This "tandem system" creates a continuous feedback loop for both the model and the workforce.

The frontier of AI training is moving beyond humans ranking model outputs (RLHF). Now, high-skilled experts create detailed success criteria (like rubrics or unit tests), which an AI then uses to provide feedback to the main model at scale, a process called RLAIF.

Using adjectives like 'elite' (e.g., 'You are an elite photographer') isn't about flattery. It's a keyword that signals to the AI to operate within the higher-quality, expert-level subset of its training data, which is associated with those words, leading to better-quality output.

The effectiveness of an AI system isn't solely dependent on the model's sophistication. It's a collaboration between high-quality training data, the model itself, and the contextual understanding of how to apply both to solve a real-world problem. Neglecting data or context leads to poor outcomes.

Do not blindly trust an LLM's evaluation scores. The biggest mistake is showing stakeholders metrics that don't match their perception of product quality. To build trust, first hand-label a sample of data with binary outcomes (good/bad), then compare the LLM judge's scores against these human labels to ensure agreement before deploying the eval.

To improve the quality and accuracy of an AI agent's output, spawn multiple sub-agents with competing or adversarial roles. For example, a code review agent finds bugs, while several "auditor" agents check for false positives, resulting in a more reliable final analysis.

A critical learning at LinkedIn was that pointing an AI at an entire company drive for context results in poor performance and hallucinations. The team had to manually curate "golden examples" and specific knowledge bases to train agents effectively, as the AI couldn't discern quality on its own.

Dr. Fei-Fei Li realized AI was stagnating not from flawed algorithms, but a missed scientific hypothesis. The breakthrough insight behind ImageNet was that creating a massive, high-quality dataset was the fundamental problem to solve, shifting the paradigm from being model-centric to data-centric.