Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Jeremy Jerry noticed Bookkeepy initially measured success by counting newly deployed AI agents, but realized that high agent volume failed to save staff hours. If team members spend more time reviewing automated campaigns than manual production requires, the setup adds friction and must be rolled back completely.

Related Insights

Before committing to automating an operational task like a daily briefing, run the process manually with AI every day for a week or two. This trial period allows you to evaluate the output's actual utility and refine the process before locking it into a potentially flawed automation.

Instead of measuring how many questions a chatbot answered, the bank tracked the reduction in support tickets for the corresponding human team, which dropped 60%. This focus on tangible business outcomes provides a much clearer picture of AI's real value than simple activity metrics.

Users often abandon AI automations at 95% accuracy because they still require manual oversight. The real value is unlocked only by investing the final effort to teach the AI and refine the process to achieve 100% reliability, truly offloading the task.

While initially a necessary safeguard, the human review process is becoming the primary factor holding back marketing execution. AI often produces better, higher-converting results faster, and the delay for manual approval negates the speed advantage.

Beyond traditional ROI, a key metric for AI success is the 'acceptance rate'—how often users accept an AI-generated output without edits. This KPI measures the tool's practical utility and trustworthiness. Defining a 'good' acceptance rate helps quantify when an AI tool is genuinely driving value versus creating rework.

When deploying an AI agent for a critical function like invoicing, it is crucial to manually walk it through the process for the first few real-world tasks. The speaker verified the agent's proposed steps and outputs for the first three deals before allowing it to run autonomously.

A Workday study reveals a critical blind spot in AI productivity metrics. While tools save time, roughly 37% of that saved time is offset by the need for rework—verifying information, correcting errors, and rewriting content. This dramatically reduces the net value and ROI of the technology.

Marketers mistakenly believe implementing AI means full automation. Instead, design "human-in-the-loop" workflows. Have an AI score a lead and draft an email, but then send that draft to a human for final approval via a Slack message with "approve/reject" buttons. This balances efficiency with critical human oversight.

Simply giving sales reps a tool that saves them 15 minutes per deal isn't enough. Leaders must proactively redesign the team's workflow, such as shifting from single-tasking to batch processing, to ensure the time saved is actually repurposed effectively.

Vanity metrics like "AI lines of code" are misleading. Coinbase measures AI success by its impact on the end-to-end development cycle: the total time from a ticket's creation to the change landing with a user. This metric holistically captures gains and focuses the team on true velocity.