AI models average their training data, resulting in generic content. This problem is compounded as new AIs train on internet data that is increasingly populated by previous AI generations' bland output, creating a self-reinforcing feedback loop of mediocrity.
Dr. O'Neil suggests some tech leaders' harmful decisions stem from misinterpreting dystopian sci-fi not as a warning, but as an exciting blueprint. They absorbed the aesthetics of works like 'Terminator' without grasping the cautionary message, leading them to build technologies that erode social contracts.
Dr. O'Neil identifies a fundamental societal divide. The empowered class sees technology as a tool for personal enhancement and productivity ('happening for me'), while the working class increasingly experiences it as an oppressive system of control ('happening to me'), fueling inequality.
Previously confined to factory floors, algorithmic monitoring like keystroke tracking is now applied to high-paid roles at companies like Meta. This signals a broad expansion of labor control, subjecting knowledge workers to the same dehumanizing surveillance long faced by hourly employees.
An algorithm is merely an expression of a company's rules and processes. Therefore, auditing an algorithm effectively means reverse-engineering the entire bureaucracy it serves—analyzing how the complete system, not just the code, treats people and distributes power.
The AI auditing field risks a race to the bottom, where firms offer cheap, superficial audits. To ensure accountability, legislation must require auditors to publicly post their methodologies and code, allowing the community to scrutinize their work and establish robust standards.
It is extremely difficult to prove a biased algorithm harmed a specific individual, as the outcome could be attributed to other factors. However, by analyzing data for an entire class of people, a clear pattern of statistical harm can be demonstrated, providing the evidence needed for legal action.
Relying on a single metric to evaluate a complex system, like the Value at Risk (VaR) model in finance, invites manipulation and disaster. Instead, systems should be managed with a 'cockpit'—a dashboard of diverse metrics that provide a holistic view and prevent stakeholders from gaming one number.
Dr. Cathy O’Neil argues the true danger of algorithms lies not in their technical sophistication but in their opaque nature, lack of oversight, and the fact that individuals cannot opt out. Even simple logistic regressions can be terrifying under these conditions.
Taylorism involved studying expert workers to codify their craft into a science owned and controlled by management. Today's AI achieves a similar end by training on expert data, concentrating knowledge and power with capital owners while devaluing individual artisan skills.
Unlike traditional ads that directly create insecurity to sell a product, platforms like X are designed to facilitate user-on-user shaming. This addictive, conflict-driven engagement is the core product, which the platforms then monetize through targeted ads from traditional 'shame profiteers'.
Dr. O’Neil proposes a framework for ethical shaming as a tool for social change. It's invalid to shame people for things they can't control (no choice) or without a chance to respond (no voice). This makes people in power, who possess both, the appropriate targets for public pressure.