Implement human-in-the-loop checkpoints using a simple, fast LLM as a 'generative filter.' This agent's sole job is to interpret natural language feedback from a human reviewer (e.g., in Slack) and translate it into a structured command ('ship it' or 'revise') to trigger the correct automated pathway.
Related Insights
Integrate AI agents directly into core workflows like Slack and institutionalize them as the "first line of response." By tagging the agent on every new bug, crash, or request, it provides an initial analysis or pull request that humans can then review, edit, or build upon.
Use a two-axis framework to determine if a human-in-the-loop is needed. If the AI is highly competent and the task is low-stakes (e.g., internal competitor tracking), full autonomy is fine. For high-stakes tasks (e.g., customer emails), human review is essential, even if the AI is good.
Instead of waiting for AI models to be perfect, design your application from the start to allow for human correction. This pragmatic approach acknowledges AI's inherent uncertainty and allows you to deliver value sooner by leveraging human oversight to handle edge cases.
Don't ask an LLM to perform initial error analysis; it lacks the product context to spot subtle failures. Instead, have a human expert write detailed, freeform notes ("open codes"). Then, leverage an LLM's strength in synthesis to automatically categorize those hundreds of human-written notes into actionable failure themes ("axial codes").
LLMs often get stuck or pursue incorrect paths on complex tasks. "Plan mode" forces Claude Code to present its step-by-step checklist for your approval before it starts editing files. This allows you to correct its logic and assumptions upfront, ensuring the final output aligns with your intent and saving time.
High productivity isn't about using AI for everything. It's a disciplined workflow: breaking a task into sub-problems, using an LLM for high-leverage parts like scaffolding and tests, and reserving human focus for the core implementation. This avoids the sunk cost of forcing AI on unsuitable tasks.
Treating AI evaluation like a final exam is a mistake. For critical enterprise systems, evaluations should be embedded at every step of an agent's workflow (e.g., after planning, before action). This is akin to unit testing in classic software development and is essential for building trustworthy, production-ready agents.
The ideal AI-powered engineering workflow isn't just one tool, but a fluid cycle. It involves synchronous collaboration with an AI for planning and review, then handing off to an asynchronous agent for implementation and testing, before returning to synchronous mode for the next phase.
An effective Human-in-the-Loop (HITL) system isn't a one-size-fits-all "edit" button. It should be designed as a core differentiator for power users, like a Head of Research who wants deep control, while remaining optional for users like a Product Manager who prioritize speed.
Define different agents (e.g., Designer, Engineer, Executive) with unique instructions and perspectives, then task them with reviewing a document in parallel. This generates diverse, structured feedback that mimics a real-world team review, surfacing potential issues from multiple viewpoints simultaneously.