Dr. Wallace distinguishes between two AI training paradigms. With supervised learning (like his ALICE bot), a creator's time is spent on 'creative writing'—manually crafting appropriate responses. In contrast, with unsupervised learning (modern LLMs), significant manual effort is spent deleting and filtering undesirable or offensive content generated by the model.
Related Insights
While AI tools once gave creators an edge, they now risk producing democratized, undifferentiated output. IBM's AI VP, who grew to 200k followers, now uses AI less. The new edge is spending more time on unique human thinking and using AI only for initial ideation, not final writing.
Generative AI is predictive and imperfect, unable to self-correct. A 'guardian agent'—a separate AI system—is required to monitor, score, and rewrite content produced by other AIs to enforce brand, style, and compliance standards, creating a necessary system of checks and balances.
The term "data labeling" minimizes the complexity of AI training. A better analogy is "raising a child," as the process involves teaching values, creativity, and nuanced judgment. This reframe highlights the deep responsibility of shaping the "objective functions" for future AI.
Richard Sutton, author of "The Bitter Lesson," argues that today's LLMs are not truly "bitter lesson-pilled." Their reliance on finite, human-generated data introduces inherent biases and limitations, contrasting with systems that learn from scratch purely through computational scaling and environmental interaction.
Instead of giving an AI creative freedom, defining tight boundaries like word count, writing style, and even forbidden words forces the model to generate more specific, unique, and less generic content. A well-defined box produces a more creative result than an empty field.
Training models like GPT-4 involves two stages. First, "pre-training" consumes the internet to create a powerful but unfocused base model (“raw brain mass”). Second, "post-training" uses expert human feedback (SFT and RLHF) to align this raw intelligence into a useful, harmless assistant like ChatGPT.
The concept of "taste" is demystified as the crucial human act of defining boundaries for what is good or right. An LLM, having seen everything, lacks opinion. Without a human specifying these constraints, AI will only produce generic, undesirable output—or "AI slop." The creator's opinion is the essential ingredient.
The transition from supervised learning (copying internet text) to reinforcement learning (rewarding a model for achieving a goal) marks a fundamental breakthrough. This method, used in Anthropic's Opus 3 model, allows AI to develop novel problem-solving capabilities beyond simple data emulation.
The best AI models are trained on data that reflects deep, subjective qualities—not just simple criteria. This "taste" is a key differentiator, influencing everything from code generation to creative writing, and is shaped by the values of the frontier lab.
Unlike traditional software, large language models are not programmed with specific instructions. They evolve through a process where different strategies are tried, and those that receive positive rewards are repeated, making their behaviors emergent and sometimes unpredictable.
