The fear of 'superhuman' AI is based on a flawed premise. Our definition of measurable intelligence—tallying numbers, memorizing lists—was created for the industrial workforce. AI is simply automating these now-outdated tasks, suggesting we need to recalibrate our measurement of human intelligence itself.

AI intelligence shouldn't be measured with a single metric like IQ. AIs exhibit "jagged intelligence," being superhuman in specific domains (e.g., mastering 200 languages) while simultaneously lacking basic capabilities like long-term planning, making them fundamentally unlike human minds.

Instead of a single, generalizable AI, we are creating 'Functional AGI'—a collection of specialized AIs layered together. This system will feel like AGI to users but lacks true cross-domain reasoning, as progress in one area (like coding) doesn't translate to others (like history).

Unlike human intelligence where skills like analytical reasoning and charisma are often decorrelated, AI systems can be trained to excel at a wide range of tasks simultaneously. General purpose learning algorithms can master both logical problems and persuasive communication, creating a more universally capable intelligence.

The goal of AI development shouldn't be to perfectly replicate human cognition, a complex and perhaps unfalsifiable target. Instead, a more pragmatic approach is to draw high-level inspiration from nature to build novel forms of intelligence designed specifically to understand and serve human needs.

Progress towards AGI is not a smooth climb. Models exhibit "spikiness"—they can perform at a world-class level on one narrow domain but degrade to a "bad high school student" with slight perturbations. This non-intuitive generalization makes their capabilities uneven and unpredictable.

The idea of a single 'general intelligence' or IQ is misleading because key cognitive abilities exist in a trade-off. For instance, the capacity for broad exploration (finding new solutions) is in tension with the capacity for exploitation (efficiently executing known tasks), which schools and IQ tests primarily measure.

Modern AIs are not programmed with explicit instructions but are trained neural nets, much like a biological brain. We cannot simply "read the code" to understand their reasoning. This "interpretability problem" is a core reason why building superintelligence is so dangerous.

Cognitive scientist Donald Hoffman argues that even advanced AI like ChatGPT is fundamentally a powerful statistical analysis tool. It can process vast amounts of data to find patterns but lacks the deep intelligence or a theoretical path to achieving genuine consciousness or subjective experience.