Language is a human-optimized construct, but the visual world is not. It contains a "fat tail" of chaotic scenes that are harder for models to learn, explaining why vision capabilities today resemble natural language processing from the GPT-3 era.

Once models reach human-level performance via supervised learning, they hit a ceiling. The next step to achieve superhuman capabilities is moving to a Reinforcement Learning from Human Feedback (RLHF) paradigm, where humans provide preference rankings ("this is better") rather than creating ground-truth labels from scratch.

OpenAI favors "zero gradient" prompt optimization because serving thousands of unique, fine-tuned model snapshots is operationally very difficult. Prompt-based adjustments allow performance gains without the immense infrastructure burden, making it a more practical and scalable approach for both OpenAI and developers.

While prompt optimization is theoretically appealing, OpenPipe's team evaluated methods like JEPA and found they provided only minor boosts. Their RL fine-tuning methods delivered vastly superior results (96% vs 56% on a benchmark), suggesting weight updates still trump prompt engineering for complex tasks.

The most fundamental challenge in AI today is not scale or architecture, but the fact that models generalize dramatically worse than humans. Solving this sample efficiency and robustness problem is the true key to unlocking the next level of AI capabilities and real-world impact.

Inspired by printer calibration sheets, designers create UI 'sticker sheets' and ask the AI to describe what it sees. This reveals the model's perceptual biases, like failing to see subtle borders or truncating complex images. The insights are used to refine prompting instructions and user training.

Fine-tuning an AI model is most effective when you use high-signal data. The best source for this is the set of difficult examples where your system consistently fails. The processes of error analysis and evaluation naturally curate this valuable dataset, making fine-tuning a logical and powerful next step after prompt engineering.

A significant real-world challenge is that users have different mental models for the same visual concept (e.g., does "hand" include the arm?). Fine-tuning is therefore not just for learning new objects, but for aligning the model's understanding with a specific user's or domain's unique definition.

When pre-training a large multimodal model, including small samples from many diverse modalities (like LiDAR or MRI data) is highly beneficial. This "tempts" the model, giving it an awareness that these data types exist and have structure. This initial exposure makes the model more adaptable for future fine-tuning on those specific domains.