Adopt a "start simple" approach for AI development. Master prompting first. If that fails, use Retrieval Augmented Generation (RAG). Fine-tuning should be the last resort due to its complexity in deployment, serving, and keeping up with rapidly evolving base models.

A Rice PhD showed that training a vision model on a game like Snake, while prompting it to see the game as a math problem (a Cartesian grid), improved its math abilities more than training on math data directly. This highlights how abstract, game-based training can foster more generalizable reasoning.

Frontier LLMs are poor tutors because they lack verifiable reward signals for learning. Brilliant's system captures real learning loops, using "did the student actually understand?" as a reward signal. This creates a unique dataset to fine-tune models specifically for tutoring.

Before considering expensive model fine-tuning, implement Retrieval-Augmented Generation (RAG). RAG dynamically retrieves information from a knowledge base to augment the prompt, solving most domain-specific problems efficiently. The recommended hierarchy is: Prompt Optimization -> Context Engineering -> RAG -> Fine-tuning.

Data that measures success, like a grading rubric, is far more valuable for AI training than simple raw output. This 'second kind of data' enables iterative learning by allowing models to attempt a problem, receive a score, and learn from the feedback.

Teams often agonize over which vector database to use for their Retrieval-Augmented Generation (RAG) system. However, the most significant performance gains come from superior data preparation, such as optimizing chunking strategies, adding contextual metadata, and rewriting documents into a Q&A format.

Static data scraped from the web is becoming less central to AI training. The new frontier is "dynamic data," where models learn through trial-and-error in synthetic environments (like solving math problems), effectively creating their own training material via reinforcement learning.

RAG systems are limited to direct retrieval and can't make spontaneous, abstract connections. This human-like ability to notice related but unasked-for concepts can only emerge from knowledge internalized within model weights, forming an associative memory.

Continuously training a model on private data internalizes concepts, reducing the need for massive context windows and system prompts. This dramatically cuts token consumption for inference compared to RAG-based approaches that re-read documents repeatedly.