The need for explicit user transparency is most critical for nondeterministic systems like LLMs, where even creators don't always know why an output was generated. Unlike a simple rules engine with predictable outcomes, AI's "black box" nature requires giving users more context to build trust.

Unlike traditional software that produces identical, auditable results, AI is non-deterministic and often can't explain its reasoning. This poses a major challenge for finance, an industry where processes must be repeatable and transparent to meet regulatory and client expectations for showing work.

To build user trust in high-stakes AI, transparency is a core product feature, not an option. This means surfacing the AI's reasoning, showing its confidence levels, and making trade-offs visible. This clarity transforms the AI from a black box into a collaborative tool, bringing the user into the decision loop.

Instead of simply providing polished answers, AI workflows should be designed to foster learning. This involves using AI to challenge an employee's hypothesis, identify weaknesses without auto-correcting, and critique reasoning, turning the tool into a coach that supports independent thought.

Unlike coding, where context is centralized (IDE, repo) and output is testable, general knowledge work is scattered across apps. AI struggles to synthesize this fragmented context, and it's hard to objectively verify the quality of its output (e.g., a strategy memo), limiting agent effectiveness.

AI agents like ChatGPT Work, built on coding assistant frameworks, are "software-brained." They focus on delivering a final product, ignoring the crucial iterative process of research, exploration, and learning that defines most knowledge work, creating a fundamental disconnect for non-coders.

Messy AI-generated code ("slop") can still result in a functional product, hiding imperfections from the end user. In knowledge work, a slightly "off" AI-generated contract or memo creates immediate legal or business risk, as there is no interface to abstract away the sloppiness.

For AI systems to be adopted in scientific labs, they must be interpretable. Researchers need to understand the 'why' behind an AI's experimental plan to validate and trust the process, making interpretability a more critical feature than raw predictive power.

AI excels at generating code, making that task a commodity. The new high-value work for engineers is "verification”—ensuring the AI's output is not just bug-free, but also valuable to customers, aligned with business goals, and strategically sound.