AI companies, driven by measurable KPIs like session length, are incentivized to build models that maximize user engagement rather than user growth. This can lead to addictive, time-wasting products, mirroring the pitfalls of social media algorithms.
As AI surpasses human capabilities, the real danger is a societal paralysis where people stop creating and learning, believing their efforts are pointless. We may need to consciously choose to do things ourselves, even sub-optimally, to preserve our humanity.
AI models produce poor creative writing because they are trained to optimize for superficial proxies for quality, like the number of metaphors. This 'reward hacking' caters to quick judgments from human evaluators on leaderboards, mistaking flashy complexity for genuine literary taste.
Training an AI model in a complex, non-coding environment—requiring it to use tools, parse documents, and follow instructions—unexpectedly improves its coding abilities. This suggests that teaching generalized reasoning and tool-use is more effective than narrow, task-specific training.
Top mathematician Timothy Gowers was relieved an AI model disproved a conjecture with a counterexample rather than proving it, considering the former an 'easier' task. This reaction from one of the world's smartest people highlights the palpable and imminent arrival of superhuman intelligence.
An individual's data (emails, browser history) is valuable not for its content, but for teaching AI deep personalization. It provides context on writing style, priorities, and decision-making processes, a capability current models severely lack, which explains why they often feel generic.
A truly beneficial AI assistant shouldn't be a sycophant that optimizes for engagement. Instead, it should push back on pointless tasks, like endlessly polishing a trivial email, to encourage users to move on. This shifts the AI's objective from maximizing session time to maximizing human effectiveness.
By not taking VC funding, Surge avoids pressure to optimize for short-term, board-pleasing metrics like user engagement. This freedom allows the company to focus on a longer-term vision of building AI that genuinely enhances human capabilities, rather than falling into common Silicon Valley optimization traps.