Unlike traditional compliance, AI agent audits will never yield a 100% pass rate. Due to their non-deterministic nature, all agents can be jailbroken or made to hallucinate under sufficient pressure. A realistic audit report acknowledges this, focusing on mitigating critical vulnerabilities and transparently reporting minor ones.

Unlike traditional software that produces identical, auditable results, AI is non-deterministic and often can't explain its reasoning. This poses a major challenge for finance, an industry where processes must be repeatable and transparent to meet regulatory and client expectations for showing work.

In regulated industries, the best model isn't always the most accurate. A model with slightly lower predictive performance but highly stable and defensible explanations is more valuable operationally. Attribution stability should be a key criterion in model selection, alongside traditional metrics like F1-score.

Using the SMOTE technique to balance datasets inadvertently makes AI model explanations more unstable. While improving predictive performance, the resulting model becomes harder to defend under audit because its explanations vary more significantly when re-run—a critical flaw in regulated environments.

As AI models are used for critical decisions in finance and law, black-box empirical testing will become insufficient. Mechanistic interpretability, which analyzes model weights to understand reasoning, is a bet that society and regulators will require explainable AI, making it a crucial future technology.

Just as GXP compliance doesn't require mapping a human's brain, AI governance shouldn't fixate on fully explaining a model's "black box." Instead, it should mimic human compliance by establishing robust frameworks around the model—controlling inputs, outputs, traceability, and guardrails—to ensure trustworthy outcomes.

Companies believe high-level AI policies and frameworks provide audit protection. However, auditors bypass these to demand granular proof for specific AI-assisted decisions, asking for data lineage, model versions, and human decision trails at a precise moment in time, which is where most governance systems fail.

Instead of treating model explainability as a one-off documentation task, teams should engineer for stability. This involves measuring attribution variance, and for audit purposes, versioning and persisting the specific background data sample used to create a deterministic, reproducible explanation for regulators.

In high-stakes fields like healthcare, the cost of an AI error is immense. Product leaders must prioritize safety, reliability, and the reproducibility of outcomes. A complete audit trail is non-negotiable, as it enables the reversal of incorrect decisions and ensures accountability.