Get your free personalized podcast brief
We scan new podcasts and send you the top 5 insights daily.
Ankur Goyal argues that AI agents can run far more exhaustive benchmarks and test more algorithms than even the best staff engineers manually could. This eliminates the common practice of prioritizing a few key benchmarks and "bullshitting" the rest, leading to more robust and performant software.
Related Insights
AMD has 'supercharged' its software development by using AI agents. These agents run in automated loops, constantly analyzing and optimizing customer models for AMD's hardware. This turns a slow, manual process into a scalable, nonstop operation, dramatically improving out-of-the-box performance for developers.
Anthropic's Claude Code team reports that AI agent skills designed for "verification"—teaching an agent to test and validate its own output—provide an extremely high return on investment. This suggests that building reliability and correctness into AI workflows is as critical, if not more so, than the initial generation capability.
Once AI coding agents reach a high performance level, objective benchmarks become less important than a developer's subjective experience. Like a warrior choosing a sword, the best tool is often the one that has the right "feel," writes code in a preferred style, and integrates seamlessly into a human workflow.
An AI agent's work output can be staggering, comparable to a high-salaried software engineer working around the clock. By simply texting instructions, a user can prompt the agent to build complex systems, generating logs that reveal an "insane" amount of published work overnight.
Most developers admit to giving pull requests only a cursory glance rather than pulling down the code, testing it, and reviewing every line. AI agents are perfectly suited for this meticulous, time-consuming task, promising a new level of rigor in the code review process.
Braintrust's CEO Ankur Goyal uses AI coding agents to solve deep technical challenges like optimizing database queries. The agents exhaustively test different solutions from database literature, a task too tedious and time-consuming for human engineers, proving AI's value on complex, high-risk problems.
While AI-powered code generation gets the attention, the most significant productivity gain for engineering teams is achieving 100% automated test coverage. This is the true unlock, as it eliminates the primary bottleneck to shipping high-quality code faster, reducing bug-fixing cycles and customer support loads.
Building a functional AI agent is just the starting point. The real work lies in developing a set of evaluations ("evals") to test if the agent consistently behaves as expected. Without quantifying failures and successes against a standard, you're just guessing, not iteratively improving the agent's performance.
AI acts as a massive force multiplier for software development. By using AI agents for coding and code review, with humans providing high-level direction and final approval, a two-person team can achieve the output of a much larger engineering organization.
An agent's effectiveness is limited by its ability to validate its own output. By building in rigorous, continuous validation—using linters, tests, and even visual QA via browser dev tools—the agent follows a 'measure twice, cut once' principle, leading to much higher quality results than agents that simply generate and iterate.
