The most effective product reviews eliminate all abstractions. Forbid presentations, pre-reads, and storytelling. Instead, force the entire review to occur within the actual prototype or live code. This removes narrative bias and forces an assessment of the work as the customer will actually experience it.
Related Insights
Don't treat evals as a mere checklist. Instead, use them as a creative tool to discover opportunities. A well-designed eval can reveal that a product is underperforming for a specific user segment, pointing directly to areas for high-impact improvement that a simple "vibe check" would miss.
Instead of guarding prototypes, build a library of high-fidelity, interactive demos and give sales and customer success teams free reign to show them to customers. This democratizes the feedback process, accelerates validation, and eliminates the engineering burden of creating one-off sales demos.
The "Owner's Delusion" is the inability to see your own product from the perspective of a new user who lacks context. You forget they are busy, distracted, and have minimal intent. This leads to confusing UIs. The antidote is to consciously step back, "pretend you're a regular human being," and see if it still makes sense.
Product teams often use placeholder text and duplicate UI components, but users don't provide good feedback on unrealistic designs. A prototype with authentic, varied content—even if the UI is simpler—will elicit far more valuable user feedback because it feels real.
When handed a specific solution to build, don't just execute. Reverse-engineer the intended customer behavior and outcome. This creates an opportunity to define better success metrics, pressure-test the underlying problem, and potentially propose more effective solutions in the future.
The common mistake in building AI evals is jumping straight to writing automated tests. The correct first step is a manual process called "error analysis" or "open coding," where a product expert reviews real user interaction logs to understand what's actually going wrong. This grounds your entire evaluation process in reality.
For complex features, a 17-page requirements document is inefficient for alignment. An interactive AI-generated prototype allows stakeholders to see and use the product, making it a more effective source of truth for gathering feedback and defining requirements than static documentation.
To keep non-technical stakeholders engaged, don't show code or API responses. Instead, have team members role-play a customer scenario (e.g., a customer service call) to demonstrate the 'before' and 'after' impact of a new platform service. This makes abstract technical progress tangible and exciting.
Instead of writing detailed Product Requirement Documents (PRDs), use a brief prompt with an AI tool like Vercel's v0. The generated prototype immediately reveals gaps and unstated assumptions in your thinking, allowing you to refine requirements based on the AI's 'misinterpretations' before creating a clearer final spec.
Instead of immediately building, engage AI in a Socratic dialogue. Set rules like "ask one question at a time" and "probe assumptions." This structured conversation clarifies the problem and user scenarios, essentially replacing initial team brainstorming sessions and creating a better final prompt for prototyping tools.