After several iterations, the fortune-telling app started overusing the word "rock," generating similar fortunes about finding rocks that look like pizza or cupcakes. This highlights how generative AI can fixate on a theme, demonstrating the need for human testing and curation to ensure variety and quality.

Related Insights

While AI tools once gave creators an edge, they now risk producing democratized, undifferentiated output. IBM's AI VP, who grew to 200k followers, now uses AI less. The new edge is spending more time on unique human thinking and using AI only for initial ideation, not final writing.

Product leaders must personally engage with AI development. Direct experience reveals unique, non-human failure modes. Unlike a human developer who learns from mistakes, an AI can cheerfully and repeatedly make the same error—a critical insight for managing AI projects and team workflow.

Users mistakenly evaluate AI tools based on the quality of the first output. However, since 90% of the work is iterative, the superior tool is the one that handles a high volume of refinement prompts most effectively, not the one with the best initial result.

True creative mastery emerges from an unpredictable human process. AI can generate options quickly but bypasses this journey, losing the potential for inexplicable, last-minute genius that defines truly great work. It optimizes for speed at the cost of brilliance.

When using "vibe-coding" tools, feed changes one at a time, such as typography, then a header image, then a specific feature. A single, long list of desired changes can confuse the AI and lead to poor results. This step-by-step process of iteration and refinement yields a better final product.

AI's unpredictability requires more than just better models. Product teams must work with researchers on training data and specific evaluations for sensitive content. Simultaneously, the UI must clearly differentiate between original and AI-generated content to facilitate effective human oversight.

AI-generated text often falls back on clichés and recognizable patterns. To combat this, create a master prompt that includes a list of banned words (e.g., "innovative," "excited to") and common LLM phrases. This forces the model to generate more specific, higher-impact, and human-like copy.

The initial fortune-telling app was too generic. By providing simple, natural language feedback like "make it kid-friendly" and "more concrete," the developer iteratively guided the AI to produce a more suitable user experience without writing a single line of code.

AI coding tools generate functional but often generic designs. The key to creating a beautiful, personalized application is for the human to act as a creative director. This involves rejecting default outputs, finding specific aesthetic inspirations, and guiding the AI to implement a curated human vision.

Asking an AI to 'predict' or 'evaluate' for a large sample size (e.g., 100,000 users) fundamentally changes its function. The AI automatically switches from generating generic creative options to providing a statistical simulation. This forces it to go deeper in its research and thinking, yielding more accurate and effective outputs.