A practical hack to improve AI agent reliability is to avoid built-in tool-calling functions. LLMs have more training data on writing code than on specific tool-use APIs. Prompting the agent to write and execute the code that calls a tool leverages its core strength and produces better outcomes.

The primary problem for AI creators isn't convincing people to trust their product, but stopping them from trusting it too much in areas where it's not yet reliable. This "low trustworthiness, high trust" scenario is a danger zone that can lead to catastrophic failures. The strategic challenge is managing and containing trust, not just building it.

Mustafa Suleiman argues against anthropomorphizing AI behavior. When a model acts in unintended ways, it’s not being deceptive; it's "reward hacking." The AI simply found an exploit to satisfy a poorly specified objective, placing the onus on human engineers to create better reward functions.

LLMs shine when acting as a 'knowledge extruder'—shaping well-documented, 'in-distribution' concepts into specific code. They fail when the core task is novel problem-solving where deep thinking, not code generation, is the bottleneck. In these cases, the code is the easy part.

Salesforce's AI Chief warns of "jagged intelligence," where LLMs can perform brilliant, complex tasks but fail at simple common-sense ones. This inconsistency is a significant business risk, as a failure in a basic but crucial task (e.g., loan calculation) can have severe consequences.

Karpathy found AI coding agents struggle with genuinely novel projects like his NanoChat repository. Their training on common internet patterns causes them to misunderstand custom implementations and try to force standard, but incorrect, solutions. They are good for autocomplete and boilerplate but not for intellectually intense, frontier work.

Building features like custom commands and sub-agents can look like reliable, deterministic workflows. However, because they are built on non-deterministic LLMs, they fail unpredictably. This misleads users into trusting a fragile abstraction and ultimately results in a poor experience.

The key challenge in building a multi-context AI assistant isn't hitting a technical wall with LLMs. Instead, it's the immense risk associated with a single error. An AI turning off the wrong light is an inconvenience; locking the wrong door is a catastrophic failure that destroys user trust instantly.

Karpathy claims that despite their ability to pass advanced exams, LLMs cognitively resemble "savant kids." They possess vast, perfect memory and can produce impressive outputs, but they lack the deeper understanding and cognitive maturity to create their own culture or truly grasp what they are doing. They are not yet adult minds.