Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Like a product requirements document (PRD), an AI skill and its evaluation (eval) are never 'done.' As you use the system, you'll learn new things. Continually ask the AI to update its own instructions to build increasingly effective automations over time.

Related Insights

The real value of custom AI skills comes from continuous refinement, not initial creation. A skill is only truly effective when it produces results that are 99% accurate with minimal human edits. This iterative process, which can take dozens of hours, is what transforms a novel tool into an indispensable workflow.

Don't just regenerate content you dislike. Provide specific feedback and then explicitly command the AI to "update the skill" with this new information. This creates a system that learns and improves from every interaction, moving beyond generating generic "lazy slop."

Static playbooks quickly become outdated. Create a dynamic 'living playbook' by having an AI agent continuously synthesize information from recent projects. It can analyze Google Docs, Slack conversations, and call notes to distill the most current best practices, ensuring your team always uses the latest version.

The highest leverage activity is creating your own skills and then providing feedback on the outputs. Instruct Claude to analyze its mistakes and rewrite the underlying skill to prevent them from recurring. This creates a powerful, compounding improvement loop.

Establish a powerful feedback loop where the AI agent analyzes your notes to find inefficiencies, proposes a solution as a new custom command, and then immediately writes the code for that command upon your approval. The system becomes self-improving, building its own upgrades.

Expect your AI agent's skills to fail initially. Treat each failure as a learning opportunity. Work with the agent to identify and fix the error, then instruct it to update the original skill file with the solution. This recursive process makes the skill more robust over time.

A truly effective skill isn't created in one shot. The best practice is to treat the first version as a draft, then iteratively refine it through research, self-critique, and testing to make the AI "think like an expert, not just follow steps."

Unlike traditional, long-lasting infrastructure, AI skills have a short half-life due to rapid model updates and changing contexts. Treat them as iterative, ephemeral assets that must be re-evaluated on a monthly basis to remain effective.

The best AI results come from iterative refinement. After an initial build, continue conversing with the agent to tweak outputs. Tell it to adjust sentence structure or writing style and redeploy. This continuous feedback loop is key to improving performance.

Instead of manually maintaining your AI's custom instructions, end work sessions by asking it, "What did you learn about working with me?" This turns the AI into a partner in its own optimization, creating a self-improving system.

Treat AI Skills and Evals as Living Documents, Not One-Time Setups | RiffOn