Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Out-of-the-box AI impresses novices but fails to meet expert standards. The product goal should be to capture an expert's knowledge through iterative feedback, so that by the third interaction, the AI's output is exceptional and personalized.

Related Insights

Developing a high-quality AI skill, like an "Ad Optimizer," is not as simple as writing a single prompt. It requires a laborious, iterative cycle of instructing, testing, analyzing poor outputs, and refining the instructions—much like training a human employee. This effort will become a key differentiator.

Unlike traditional software where problems are solved by debugging code, improving AI systems is an organic process. Getting from an 80% effective prototype to a 99% production-ready system requires a new development loop focused on collecting user feedback and signals to retrain the model.

The real value of custom AI skills comes from continuous refinement, not initial creation. A skill is only truly effective when it produces results that are 99% accurate with minimal human edits. This iterative process, which can take dozens of hours, is what transforms a novel tool into an indispensable workflow.

Users mistakenly evaluate AI tools based on the quality of the first output. However, since 90% of the work is iterative, the superior tool is the one that handles a high volume of refinement prompts most effectively, not the one with the best initial result.

Getting a useful result from AI is a dialogue, not a single command. An initial prompt often yields an unusable output. Success requires analyzing the failure and providing a more specific, refined prompt, much like giving an employee clearer instructions to get the desired outcome.

As AI capabilities become commoditized, the key to superior output is the user's domain expertise. An expert with precise vocabulary can guide an AI to produce better results in one attempt than a novice can in many, because they can articulate the desired outcome more effectively.

A truly effective skill isn't created in one shot. The best practice is to treat the first version as a draft, then iteratively refine it through research, self-critique, and testing to make the AI "think like an expert, not just follow steps."

Since current AI is imperfect, building for novices is risky because they get stuck when the tool fails. The strategic sweet spot is building for experts who can use AI as a powerful but flawed assistant, correcting its mistakes and leveraging its strengths to achieve their goals.

The best AI results come from iterative refinement. After an initial build, continue conversing with the agent to tweak outputs. Tell it to adjust sentence structure or writing style and redeploy. This continuous feedback loop is key to improving performance.

Treat AI skills not just as prompts, but as instruction manuals embodying deep domain expertise. An expert can 'download their brain' into a skill, providing the final 10-20% of nuance that generic AI outputs lack, leading to superior results.

Aim for AI's Third Output to Impress Experts, Not the First | RiffOn